PyCon and Adding Maintainers to Click

written by David Lord on 2018-06-01 in Meta

In May I attended PyCon US 2018 in Cleveland, Ohio. It's a great opportunity to meet people interested in the Pallets projects and Python web applications. It can be a little intimidating, but also a lot of fun. The biggest difficulty is that there are too many things to do, and I always wish everyone could have a few more days together.

During the main conference, while talks are happening, anyone can organize "open spaces," impromptu meetings for any topic. I organized a Pallets open space, and we had a huge turnout. I was kind of unprepared for the size, so we started with people asking me questions, and then split up into tables for new users, Flask, Click, and Jinja.

Open space announcement

For up to four days after the talks, developers get together for sprints, working together to contribute to Python open source projects. Experienced developers help new ones learn how to contribute, and everyone gets to learn about new projects and meet the developers behind them.

Pallets open space

At the open space, I had mentioned that no one had really been maintaining Click for the past year. It turns out a lot of people use Click. They came motivated from the open space to contribute during the sprint. Some were new to contributing, but they all had one thing in common: they knew how Click was being used more than I did! So I took a chance and gave them all write permissions to the repository. Click started at ~250 open issues and 68 pull requests. After 2 days, it was at 140 issues and 22 pull requests. Wow! With that success, I'm officially welcoming all these new maintainers to the Pallets Click team. While Click stole the show, we had great contributions to Werkzeug, Flask, and Jinja as well.

Dinner with Pallets sprinters

If you want to get involved, a great way is to watch our repositories on GitHub. You'll get notifications for each issue, so you can see what's happening and start contributing. You can help by triaging issues, improving tests and documentation, and fixing bugs. If your company uses a Pallets project like Flask, Click, or Jinja, consider pointing them to our PSF donation page. Donations will help get maintainers to more events so we can do more sprints like this one. Thank you to everyone at PyCon and in the community for making Pallets a success!

Donate to Support Pallets

written by David Lord on 2018-04-26 in Meta

Pallets is excited to announce that we have joined the Python Software Foundation's Fiscal Sponsorship program. As a non-profit organization, the PSF will accept donations on behalf of Pallets into a dedicated account.

Donations enable more attention from maintainers, which translates into more time devoted to fixing bugs, developing features, and making quicker releases. Click here to donate today.

The Pallets organization develops and supports Flask, Jinja, Werkzeug, Click, and other Python libraries. These libraries power applications of all sizes around the world, and are downloaded millions of times each month. Despite their popularity, the projects are primarily maintained by only a few developers. The goal of Pallets is to grow the community around these projects to create a sustainable group of contributors and users.

Your donation will help:

  • Allow maintainers to devote more of their time to the projects.
  • Recognize outstanding contributors in the community.
  • Allow maintainers and community members to attend conferences.
  • Acquire infrastructure and developer tools.
  • Sponsor local meetup events.
  • And more!

If you like the work we do, you may donate as an individual. If your employer uses Flask or any of the Pallets projects, reach out to them to donate.

Flask and the Pallets team depends on you, the community. We thank you for all the contributions you've already made, and look forward to growing the community even more.

Click here to donate today.

Flask 1.0 Released

written by David Lord on 2018-04-26 in Security , Releases

The Pallets team is pleased to release Flask 1.0.

The Flask framework has been stable for a long time. A little more than 8 years after the first commit, the version number finally reflects that. 1.0 comes with a significant number of changes representing over a year of work.

  • Dropped support for Python 2.6 and 3.3.
  • The CLI is more flexible. FLASK_APP can point to an app factory, optionally with arguments. It understands import names in more cases where filenames were previously used. It automatically detects common filenames, app names, and factory names. FLASK_ENV describes the environment the app is running in, like development, and replaces FLASK_DEBUG in most cases. See the docs to learn more.
  • If python-dotenv is installed, the flask CLI will load environment variables from .flaskenv and .env files rather than having to export them in each new terminal.
  • The development server is multi-threaded by default to handle concurrent requests during development.
  • flask.ext, which was previously deprecated, is completely removed. Import extensions by their actual package names.
  • Accessing missing keys from request.form shows a more helpful error message in debug mode, addressing a very common source of confusion for developers.
  • Error handlers are looked up by code then exception class, on the blueprint then application. This gives more predictable control over handlers, including being able to handle HTTPException.
  • The behavior of app.logger has been greatly simplified and should be much easier to customize. The logger is always named flask.app, it only adds a handler if none are registered, and it never removes existing handlers. See the docs to learn more.
  • The test_client gained a json argument for posting JSON data, and the Response object gained a get_json method to decode the data as JSON in tests.
  • A new test_cli_runner is added for testing an app's CLI commands.
  • Many documentation sections have been rewritten to improve clarity and relevance. This is an ongoing effort.
  • The tutorial and corresponding example have been rewritten. They use a structured layout and go into more detail about each aspect in order to help new users avoid common issues and become comfortable with Flask.

There are many more changes throughout the framework. Read the full changelog to understand what changes may affect your code when upgrading.

JSON Security Fix

Flask previously decoded incoming JSON bytes using the content type of the request. Although JSON should only be encoded as UTF-8, Flask was more lenient. However, Python includes non-text related encodings that could result in unexpected memory use by a request.

Flask will now detect the encoding of incoming JSON data as one of the supported UTF encodings, and will not allow arbitrary encodings from the request.

Install or Upgrade

Install from PyPI with pip:

pip install -U Flask

Get Involved

Flask and the Pallets team depends on you, the community. Whether you report issues, write documentation, create patches, or answer questions, we appreciate all the help you provide. Check out the contributing guide to get started.

The Pallets organization has joined the Python Software Foundation's Fiscal Sponsorship program. We now accept donations through the PSF in order to support our efforts to maintain the projects and grow the community. Click here to donate.

Flask 0.12.3 Released

written by David Lord on 2018-04-26 in Releases , Security

This release includes an important security fix for JSON and a minor backport for CLI support in PyCharm. It is provided for projects that cannot update to Flask 1.0 immediately. See the 1.0 announcement and update to it instead if possible.

JSON Security Fix

Flask previously decoded incoming JSON bytes using the content type of the request. Although JSON should only be encoded as UTF-8, Flask was more lenient. However, Python includes non-text related encodings that could result in unexpected memory use by a request.

Flask will now detect the encoding of incoming JSON data as one of the supported UTF encodings, and will not allow arbitrary encodings from the request.

Upgrade

Upgrade from PyPI with pip. Use a version identifier if you want to stay at 0.12:

pip install -U Flask==0.12.3

Or upgrade to 1.0:

pip install -U Flask

Werkzeug 0.14 Released

written by Armin Ronacher on 2017-12-31 in Releases

The Pallets team is pleased to release Werkzeug 0.14. Changes include:

  • Improved the usefulness of Request.application by automatically handling HTTP exceptions.
  • Added support for platforms that lack SpooledTemporaryFile. This primarily affects GAE users which were unable to use 0.13 due to this missing API.
  • Add support for etag handling through if-match
  • Added support for the SameSite cookie attribute along with better support for invalid cookies.
  • Added a HTTP proxying middleware (werkzeug.wsgi.ProxyMiddleware)
  • Various improvements for the reloader.
  • The built-in HTTP server will no longer close a connection in cases where no HTTP body is expected (204, 204, HEAD requests etc.)
  • Werkzeug will no longer send the content-length header on 1xx or 204/304 responses.
  • Added support for static weights in URL rules.
  • Better handle some more complex reloader scenarios where sys.path contained non directory paths.

Read the full changelog.

Install or upgrade

Install from PyPI with pip:

pip install -U Werkzeug

Get Involved

Werkzeug and the Pallets team depends on you, the community. Whether you report issues, write documentation, create patches, or answer questions, we appreciate all the help you provide. We updated the contributing guide to help make it easier to get started.

Werkzeug 0.13 Released

written by David Lord on 2017-12-07 in Releases

The Pallets team is pleased to release Werkzeug 0.13. Changes include:

  • Deprecated Python 2.6 and 3.3 support. The next version, 0.14, will remove support.
  • The built-in development server supports receiving requests with Transfer-Encoding: chunked.
  • Header options without an extended encoding can contain single quotes. For example, filename=t'es't.txt is now valid.
  • Removed werkzeug.script in favor of Click.

Read the full changelog.

Install or upgrade

Install from PyPI with pip:

pip install -U Werkzeug

The PGP key ID used for this release is David Lord: 43368a7aa8cc5926.

Get Involved

Werkzeug and the Pallets team depends on you, the community. Whether you report issues, write documentation, create patches, or answer questions, we appreciate all the help you provide. We updated the contributing guide to help make it easier to get started.

Security bugs on Windows servers: Flask 0.12.2 and Werkzeug 0.12.2 released

written by Markus Unterwaditzer on 2017-05-16 in Releases , Security

Flask 0.12.2 and Werkzeug 0.12.2 have been released. They contain the same security bugfix for the safe_join function in each package.

The problem only occurs if you are running your application on a Windows server.

Details

David Lord initially found this bug (thanks!) and disclosed it to the other maintainers in a private email:

While going through PR #2059 about safe_join, I looked up Python's ntpath.join and discovered a vulnerability that safe_join on Windows doesn't cover.

https://docs.python.org/3/library/os.path.html#os.path.join: "os.path.join("c:", "foo") represents a path relative to the current directory on drive C: (c:foo)"
safe_join('\\root\\path', 'd:', 'test.txt') would break out of the trusted root directory and instead take the test file relative to the cwd on the d drive. This doesn't give completely arbitrary path access, since it's limited to the cwd, but it's still not good.

For the application developer this means that endpoints using safe_join could potentially be used to disclose arbitrary files in the server processes' current working directory on Windows.

What happens next

We strongly recommend upgrading to Flask 0.12.2 and Werkzeug 0.12.2, as this bug has been fixed there (Flask, Werkzeug).

A CVE has been requested on Tue, 16 May 2017 06:51:09 +0000, the CVE CVE-2017-9088 was assigned.

Werkzeug 0.12 released

written by Markus Unterwaditzer on 2017-03-10 in Releases

We just released Werkzeug 0.12. Highlights from the changelog:

Further deprecation of Werkzeug's script module

Werkzeug's script module was deprecated since 2011. Deprecation slipped through the cracks, so we now additionally show large warnings at runtime. Head over to the bugtracker for the rest of the plan.

Defaults for password hashing changed

Password hashing now uses sha256 by default instead of sha1. On a related note, certificates generated with the dev server are now signed with sha256 instead of md5.

Jinja 2.9 Released

written by Armin Ronacher on 2017-01-07 in Releases

After a long time of no changes a new release, 2.9 codename “Derivation” of the Jinja template engine for Python is out. This change is probably the biggest release in Jinja2's history and it requires a bit of explanation of why it happened and why it happened now. But first let's cover the big changes in it.

New Identifier Tracking

This appears to be an under the hood change because it effectively just changes the internal algorithm that Jinja2 uses to map it's own scoping rules onto the Python interpreter's scoping rules but it has wide ranging consequences. The original tracking issue goes back to 2011 and was created as a long running TODO item. There were a few reasons why it was implemented this way but over the years it became clearer and clearer that the method chosen for mapping identifiers is just causing too many issues. The reason it was not changed was that the worries were too big that people accidentally relied on such bugs.

However with improved support for pinning to older versions of Python packages and the fact that people no longer use linux distribution provided packages much the worries about greater changes are heavily reduced. With Jinja 2.9 the identifier tracking was completely changed which should squash pretty much all the weird behaviors developers might have encountered and also enabled support for related improvements. In particular the new identifier tracking generalized the context behavior for includes and imports.

In the ideal case you won't notice anything other than that some constructs are possible now that caused errors in the past.

Python 3 Feature Support

While Jinja2 supported Python 3 for years now it never really embraced functionality that the language provides on 3.x that it does not do on 2.x. Largely that is caused by the fact that we want templates to be compatible between 2.x and 3.x (this is true only for as long as template variables are restricted to ASCII characters). However 3.6 now added async generators which permits Jinja2 to fully support the async and await keywords on 3.6 and later.

In particular it means that you can now return coroutines from functions passed to Jinja2 templates and the template engine will automatically await them. Likewise all filters were updated to work with iterators as well as async iterators alike.

Additionally Jinja2 now emits generator_stop as a future flag which means that accidentally emitting StopIteration from a filter or other code will no longer cause the template to just stop rendering silently. This was enabled by PEP 479.

Policy Framework

Jinja2 now has an internal policy configuration which permits one to centrally reconfigure behavior of filters and other things. While so far not much can be reconfigured it will greatly support future improvements. This also finally allows us to provide a default tojson filter which previously was only available through Flask or other frameworks. Through the policy configuration the particular form of JSON serialization can be customized and replaced.

Full Changelog

  • Change cache key definition in environment. This fixes a performance regression introduced in 2.8.
  • Added support for generator_stop on supported Python versions (Python 3.5 and later)
  • Corrected a long standing issue with operator precedence of math operations not being what was expected.
  • Added support for Python 3.6 async iterators through a new async mode.
  • Added policies for filter defaults and similar things.
  • urlize now sets "rel noopener" by default.
  • Support attribute fallback for old-style classes in 2.x.
  • Support toplevel set statements in extend situations.
  • Restored behavior of Cycler for Python 3 users.
  • Subtraction now follows the same behavior as other operators on undefined values.
  • map and friends will now give better error messages if you forgot to quote the parameter.
  • Depend on MarkupSafe 0.23 or higher.
  • Improved the truncate filter to support better truncation in case the string is barely truncated at all.
  • Change the logic for macro autoescaping to be based on the runtime autoescaping information at call time instead of macro define time.
  • Ported a modified version of the tojson filter from Flask to Jinja2 and hooked it up with the new policy framework.
  • Block sets are now marked safe by default.
  • On Python 2 the asciification of ASCII strings can now be disabled with the compiler.ascii_str policy.
  • Tests now no longer accept an arbitrary expression as first argument but a restricted one. This means that you can now properly use multiple tests in one expression without extra parentheses. In particular you can now write foo is divisibleby 2 or foo is divisibleby 3 as you would expect.
  • Greatly changed the scoping system to be more consistent with what template designers and developers expect. There is now no more magic difference between the different include and import constructs. Context is now always propagated the same way. The only remaining differences is the defaults for with context and without context.
  • The with and autoescape tags are now built-in.
  • Added the new select_autoescape function which helps configuring better autoescaping easier.

Why an Update now?

One of the reasons Jinja2 was lying largely dormant is that it's always scary to touch something so many people use. While bugs were known for quite some time it requires careful changes to not change all the template code out there. In particular once salt and ansible started using Jinja it became a big emotional burden to do changes on it.

However it turns out that emotional burden does not get smaller for as long as those bugs are in there. Going in and cleaning up the behavior actually turns out to be healthier in the long run. Now even if this release introduces regressions in behavior, the internal code quality is much improved and reasoning about the runtime and compiler is a lot easier now.

On a personal note I'm happy to see how popular Jinja has become and I hope that with this release it becomes more enjoyable to write Jinja templates.

Aside: if you are downloading Jinja2 you will actually pull 2.9.1 because there was a regression in 2.9 that was only found after releasing it. In an ironic twist it was found when this website attempted to render a complex template with Jinja 2.9 on the build server.

Jinja 2.8.1 Security Release

written by Armin Ronacher on 2016-12-29 in Releases , Security

We just pushed out a new release for Jinja (2.8.1) which includes a security related fix. If you are using the Jinja2 sandbox you are encouraged to upgrade or alternatively manually further lock down the sandbox.

The core of the issue is that Python's string format method that was added to strings can be used to discover potentially dangerous values including configuration values:

>>> config = {'SECRET_KEY': '12345'}
>>> class User(object):
...  def __init__(self, name):
...   self.name = name
...
>>> user = User('joe')
>>> '{0.__class__.__init__.__globals__[config]}'.format(user)
"{'SECRET_KEY': '12345'}"

For this reason you must never let user supply format strings in raw Python as its too easy to escape them. However specifically for the Jinja2 sandbox we changed the behavior now that we're using the same sandboxing functionality that Jinja2 uses for its own runtime also for Python string formatting.

This means that with 2.8.1 and higher templates from sandboxed environments will intercept format strings the same way as with normal cases:

>>> from jinja2.sandbox import SandboxedEnvironment
>>> env = SandboxedEnvironment()
>>> class User(object):
...  def __init__(self, name):
...   self.name = name
...
>>> t = env.from_string(
...  '{{ "{0.__class__.__init__.__globals__}".format(user) }}')
>>> t.render(user=User('joe'))
Traceback (most recent call last):
  ...
jinja2.exceptions.SecurityError: ...

If you don't want or you cannot upgrade Jinja2, you can override the is_safe_attribute method on the sandbox and explicitly disallow all format attributes on strings.

Thank you to Olivier Dony for reporting the issue.