one current problem for core-devs seems to be the large number of PRs that they have to review. Besides basic correctness, they need to ensure that these PRs do not have unintended consequences, and that they will not become a maintenance problem.
However, a bottleneck at this step would slow down the progress of new features and discourage contributors.
I want to propose that a new repo (or a sub-package within the main numba package) is created for addons. These are ideas which might be good, but maybe they don’t belong in the main package, or maybe they could become a problem and it should be clear that core-devs are not responsible for support and maintenance.
Periodically, those features that are popular and stable enough, could be integrated in the main repo. I think Tensorflow did something similar with the tensorflow.contrib package.
very interesting read, I knew contrib turned into addons but I was not aware of this document.
It seems that TF had a "contrib model` where community contributions :
had core-devs oversight, including reviews and taking over maintenance.
were distributed with the core package
Now they moved to the “addons model” where
contributions live in a separate repo, each managed by a different group of users
with some TF central team help to setup build and release workflow.
My original proposal is open to both models (with a caveat described below). Each has pros and cons.
I would point out that the fact that TF moved to the addons model does not mean Numba should not consider the contrib model. After all, they do say
tf.contrib module plays several important roles in the TensorFlow ecosystem
and
as the community has grown, the lack of scalability […] has become apparent.".
I’m reading that as “contrib was good when we were small, but it didn’t scale”. But I might be reading it wrong.
The size of Numba’s community is probably closer to early TF than current TF. So I think the contrib model could be considered with one caveat: in tf.contrib the reviews were performed by core-devs, and the key part of my proposal is to ease the review (and support) burden on Numba’s core-devs. As @EPronovost correctly points out " the tensorflow model […] may not actually make for less work overall.". We would need to tweak that to make it work for Numba.
Yes I think we’d have to figure out what role a contrib module would serve for numba. Given all the discussion about stability and usability, I’m not sure if having a module that’s not as tested/documented/maintained would align with that goal.
You have a point on stability and usability. In that regard the addons model is better, since it’s less likely that an inexperienced user would use an add-on if they don’t come in the same package as core Numba. And it’s easier to understand that addons are not maintained to the same standards as core Numba.
I don’t have a rock solid opinion on an add-on repo, since I see it’s incredible usefulness but also the maintenance burden it can bring.
Yet I think it would definitely be useful to have a central point where people can share neat workflows/ tricks/ snippets. Often, clever things are “developed” in a discussion and if people have the energy to write it up nicely (e.g. markdown or sphinx style), one could develop some kind of user compendium. To clarify: I am thinking of things that do not necessarily turn into fully implemented packages or addons but rather certain design patterns etc. And if they eventually become outdated they can be deleted or marked as defunct without putting dependency chains at a big risk. Maybe this could even just be a specific section of this messaging board where “finalised” solutions live, so that people don’t have to read through whole threads of discussion?
I agree, I’ve been thinking along the same lines. Even though I still think the addons repo is useful, there’s also a lot of good stuff coming from gitter (and now discourse). Last week I opened https://github.com/luk-f-a/numba-how-to, which will be an experiment in collecting useful recipes that I find here (and ofc, contributions are welcome). For small scale code, this should be a better venue than a full repo, because it’s a documentation only project.
Compared to a discourse page, I’ll try to make it easier to retrieve by categorizing them.