It is common knowledge that pickle is a serious security risk. And yet, vulnerabilities involving that serialisation format keep happening. In the article I shortly describe the issue and appeal to people to stop using pickle.
You must log in or # to comment.
The thing is, none of the suggested alternatives can do what pickle does, and the article focuses on a narrow (albeit ubiquitous) use case: serialisation of untrusted data.
There are still legitimate use cases for pickle, especially when storing, caching, or comparing objects that can’t easily be serialised with say, JSON or TOML. It’s a question of using the right thing for the right job is all, and pretending like JSON is a comparable alternative to pickle doesn’t help anyone.
If you’re serialising trusted data, you can define schema for it and use Protocol Buffers which will not only by safer but also faster. Pretending that you need to be able to serialise arbitrary data hurts everyone.
Also there is strictyaml that validates against schemas. Don’t touch the builtin yaml module.
protobuf needs to be compiled. This introduces possibility of coder error. Just forgetting to compile and commit protobuf files after a change. This affected the electrum btc and ltc (light) wallets.
Also there is strictyaml that validates against schemas. Don’t touch the builtin yaml module.
Thanks. I’ll include that in an update.
protobuf needs to be compiled. This introduces possibility of coder error. Just forgetting to compile and commit protobuf files after a change. This affected the electrum btc and ltc (light) wallets.
Yes, that’s certainly a downside. It also demonstrates one should not commit such generated files. A better approach is to commit the source files (in this instance message definition) and have a compilation step included in the program’s build/install recipe.
strictyaml
A better approach
That unfortunately isn’t a better approach. The compilation step requires protobuf to be installed, by the distro package manager. To my knowledge it’s not available from pypi.
An uncompiled protobuf file is essentially worthless unless it’s compiled. But if it’s compiled then it’s a binary blob.
Not anti-protobuf. Just make the protobuf compiler available without getting a distro package manager involved.
Otherwise slower alternatives might be more viable.
strictyaml bundles strictyaml.ruamel, which used to be an external unmaintained C package.
This reduces strictyaml dependencies to:
pyproject.toml
dependencies = [ "python-dateutil>=2.6.0" ]Just that one. So can be confident strictyaml will work.
Can the same be said for protobuf and Google (over invested in AI and is probably dying underneath a huge debt burden while spending tons of money on AI wash propaganda while not funding Python projects enough. Maintainer leave or burn out while everyone is too busy head fcking us with the AI washing to notice.)
It is a better approach, it just may be more complex. Only people developing or packaging the library need to compile the message definitions. It’s not a big burden to require than they have
protocinstalled. The end user will only need to depend on the created package.It’s a potential single point of failure. Which have experienced first hand. The rest of the app could not run cuz a non-essential piece was non-operable due to the missing compiled message definitions file or message definitions file was updated but not compiled.
So protobuf carries a non-zero risk.
Could the app have been designed without an essential exploding binary blob? Most definitely yes!
Writing software carries a non-zero risk. If compiling was part of building the package rather than manually committed to the repository, things would work. And that would make the design have no essential binary blob.
I second this.
If you need to
pickleyour ML model, just use JobLib instead.If you want to save a polars or pandas df, save files as parquet.
Both ways you can also use compression, so you’ll save space as well. Use
zstdif you need decent compression, orlz4if you write and read speeds.Joblib has the same drawback as
pickle. From the documentation:joblib.dump() and joblib.load() are based on the Python pickle serialization model, which means that arbitrary Python code can be executed when loading a serialized object with joblib.load().
joblib.load() should therefore never be used to load objects from an untrusted source or otherwise you will introduce a security vulnerability in your program.
