Friday, October 18, 2013

Team Modernize

&TLDR;

Have you wanted to contribute to a major Plone release but felt nervous or perhaps feeling a bit like you don't have the chops (aka impostor syndrome)? Can you spare a 1-4 hours? Have I got a deal for you!

Details

It has been a VERY busy summer and I may have disappeared for a bit to actually make money but I'm getting back into the swing of things and rediscovering a PLIP I proposed, thanks to some lovely nagging by @bloodbare.

 In one year, many things have changed and Plone 5 is real. We started at the Plone conference in Arnhem and things just lagged - MY BAD! The problem with this PLIP was that it is a bit overwhelming and I just couldn't get motivated to continue. To be fair no one else did either because its a mega poopy task. It was difficult to track and the branches were dizzying. Lucky for us all, we can move forward differently in a way that is a bit cleaner.

I have carefully broken apart this PLIP* into a lots of tiny little pieces. Each task should be around 1-4 hours. Some may even be shorter than that. If you want to help with this PLIP all you have to do is:

  1. Get the Plone 5 buildout running on your machine. This is the master branch (it's so exciting!). If you have not signed a contributor agreement yet don't let that stop you. Send it my way to assignments@plone.org and we will get you started.
  2. Accept a ticket from the list of baddies. Some tickets are literally just "Delete these files and make sure no tests break". Yep. Get those before they are gone!
  3. Modernize like mad
  4. Run test cases locally
  5. Commit the changes and make a pull request. Make sure to indicate the ticket number in the request. The pull requests go to master branch.
  6. Make sure to followup with jenkins and the other robots to verify you didn't break the build. In theory they actually yell at you!
  7. GOTO 2**

And that's it! For your help you will get:

  1. My undying admiration and appreciation
  2. A chance to participate in a major open source software release (very cool stuff)
  3. Upon Plone 5 launch, a t-shirt, laptop sticker, or fez celebrating the release of Plone 5 and indicating that you were a core contributor in the decrufting of this bad boy. ***
  4. The chance to save that dirt old feature you love, or remove that buggy old feature you hate
  5. A chance to go through the pull request cycle with a group of people who are committed to be very sweet while giving you code review and pointers. What other PLIP can guarantee that???
I am here to help through the whole process. If you are interested and still don't know where to start, just find eleddy on IRC and I'll walk you through it. These tickets are a once in a lifetime way to get involved in a major framework and a major release.

I can't do it alone, Ploners. Together, we can decruft this bad boy all the way to Plone 5. 



* There are indeed still some tickets remaining to be made and I promise I'll get to those soon! That doesn't mean I don't need help now!
** or not. Seriously, if you just picked one ticket I would still jump for joy!
*** I promise to think of something clever

Friday, June 14, 2013

Operation Flawless Login

No booze, no caffeine, training,
bad wifi, jetlag, and 100+ users
on every type of device under
the sun who couldn't reliably
log onto the system. PAIN.
I was going to do the whole play it cool thing and wait 3 days for the next post but I am just so excited about this new "blogging" trend. I'm hoping to make it a Friday tradition (preceded by 2 cups of drip coffee and followed by a nap).

One thing I'm super excited to start working on as part of the migration away from cpy/cpt is redoing login. As far as Plone goes, you are likely in 1 or 2 camps: 
  1. Works fine for me!
  2. Rot in hell you !@#$ old code
I suspect most of you are in camp 1, or just have no opinion whatsoever so I'm hoping I can convince you to jump over to camp 2. Most of the following was originally written with an intense amount of pain (see photo) so if it seems like I'm shouting at times its because I was: I probably qualified as temporarily insane. Here we go!

Ryan Foster (@dextermilo) recently proposed this beautiful, simple, login page for a client which has all css/js/etc embedded making it load very fast (1 template + logo). Tell me you don't want this login page HARD. I know I do.

Login as a First Impression

For sites that are completely blocked off by default, login is the first experience with a new system/site. For anyone who is invited to a system, its the same case. In fact, I would argue that a seamless login is the most important introduction to any web based system. It should be flawless. Absolutely flawless. 

And this is where "we the Plone" fall flat on our face. A sampling of the issues, mostly focused on the ones dear to my heart at the moment:
  • The login, registration, password reset pages are too slow from relying on a framework that it doesn't need, causing more complications with frustrated users. We can't login 100 people at once even with load balancing. Trainers around the world know exactly what I'm talking about. How can you ramp up a system that people can't login to fast and accurately?
  • We are 100% not ready for mobile. There are some issues with random phones that we will never be able to address but we also need to catch up with HOW people use mobile devices. The best example is the shitty network experience. What happens when you click on a  registration link and the network goes out before you can complete? You go back to the email invite and RE-CLICK. But wait… whats that? Plone only allows you to click once and the link expires? 1999 called and wants their user restrictions back. 
  • The messaging/wording of invite and password reset emails is robotic and offers no indication to what the user just signed up for or were invited to. They get the email and have questions like "Wait, so how to I get to the home page?" and "What is my username?" and "Who is admin?". If you have signed up for basically any service at any point lately I'm sure you know what a great invite email feels like.
  • Testing? What's that? 
  • Have you seen this code? I mean seriously (WARNING: NSFW) look at that shit. It's 10 years of whacked out random bug fixes that no one has any idea if its ok to rip out or not. Its 20+ different files spewed throughout the code in portal_skins style that requires a diagram the size of my kitchen table to print out. Bugs languish in the tracker for years because everyone is terrified to touch it. It is officially unmaintainable code. Don't get me wrong, it was great in its hay day. But I'm sorry login_form.pt, it's time for us to go our separate ways.
  • Customization. Ugh. Enough said.
  • There is no OOB way to block off a site.
  • and so much more… I don't have time to list them all. Have a favorite login gripe? Comment below!
Summary: It's a liability to Plone and more importantly my projects, and my job these days is to minimize liability. I'm coming for you login!

Battling the login monster one head at a time takes too long. Instead we cut out the heart, then roast the heads in olive oil for 8 hours, top with a dollop of creme freche and a sprinkle of truffle oil. Serve immediately with a spicy Triple.

I Love You but I've Chosen Rewrite

In good faith of open source complaining, I am determined to help fix it. In fact, Matt Hamilton has already agreed to help and get this handled and I secretly suspect that he has more deep down pain than I since he is taking off with it and already planning a quick weekend sprint to kick it off. Are you one a member of the login burned? Help us out!

Here is a brain dump of the things I personally want to accomplish with login rework:
  • Login gets dirt fast. It should handle 100 simultaneous logins on 1 site, 1 box, standard hardware. I want to say 300 but 100 is a good target and we can't have mass training sessions without this. I think a lot of the other options can help on this. I am tired of pointing to caching as the solution. There are a lot of transaction issues in this but we have to start somewhere. The skeletons oh the skeletons!
  • Stop basing on main template (this also applies to 404, 500, etc). If a site is closed off and resources were just recooked, thats 20+ resources the browser has to load and thats the immediate impression for the users. Get this down to less than 3 files (a login splash type thing, ideally 1 page with embedded resources). Ability to customize VERY easily (diazo?) but more importantly having a good default. 
  • Mobile. Responsive, quick. Avoid redirects at all costs and where they are make it sane. I recently saw password reset on initial login fail on a variety of mobile devices. We are doing something wrong/old school/something. 
  • Integrate iw.rejectanynmous to core. 
  • Sane invite messages. Default system invite messages are horrid. They don't have site urls, nor indicate the username or who or what or… anything really. Ideally this message can be customized TTW (e.g. welcome to the site. this is our new intranet and I want you to use it like x,y,z your login is x the url is y and I am your new overlord)
  • Forgotten password. Oh god. Expire on click is a disaster, and the methodology is generally buggy.
  • Tests. Tests everywhere.  I want robot going crazy too.
  • Security. Lots of it.
  • Login/reset/all forms rewritten as z3c. This allows us to extend and add things like site agreements. 
  • Simplicity. There are too many options to be sane at the moment. We need to not even look at the code, and think "what do people REALLY need to configure in login" (especially that can't be done in PAS). We need to worry about our 80% use case first. I will be rabid about this. Login, login, login. And that's it. For now.
  • Rip out openid as it is. A simple pluggable solution is the answer to proper integration.
  • Default captcha for open registration.
  • Integrate some of the login api work that has been done. 
And on my "after that is done" wish list:
  • Must reset password is weird and has never worked properly as I have wanted it implemented. The ability to expire passwords from some control panel has been on a wish list for many clients over the years. Time based expiration as well. That works, not that is a hack into the current whacky code base.
  • Tracking for site admins. Who hasn't logged in in 2 weeks? 1 month? Who has NEVER logged in? How can I make sure people are actually using the site? What kind of data do we need to capture to answer security compliance questions? We can't hide behind LDAP integrations for that forever. Also not first pass, but the ability to generate these reports must be considered.
In the category of "really dreamy some day if I can find the time":
  • Plone as oauth consumer. It's 2013 and "login with Facebook" isn't going away. *sigh*. I don't know that this should be in core but the ability to do setup oauth is HAWT.
  • Plone as an authentication provider. 
We are ripping out portal_skins piecemeal so that we can get the new stuff out to integrators ahead of time to test then merge it back into core for the release. Login is its own package, plone.future.login and anyone is welcome to help the rewrite. The theme is release early, release often, simple, fast, extensible.

Easy enough right?

Tuesday, June 11, 2013

PSM 2013 Days 1-4: Everything you Never Wanted to Know About Plone 5


My, my, my it's been a while Ploners. I deeply apologize for lack of communication on my part and my only excuse is laziness, impatience, and hubris.

How does one sum up the realization that communication in the Plone community has hit rock bottom? One can't. So this will likely be the longest blog post I ever write and hopefully the last one of this length. Anyone who has ever said they get pissed that "we" don't communicate will leave this post with sincere regrets about asking for more info. If you are lucky I will put a TOC on this manifesto, but don't hold your breath. It's technical, it's marketing, and everything in between. Grab a cup of coffee and join me on the 7 day adventure that was Plone Symposium Midwest 2013 (PSM13).

Technical note: unlike certain content management systems like Plone, blogger does not have pull quotes. I'll do my best to mimic them for you lovelies, but I'm limited here. 

Pre Sprint


Pre-sprinters at the park
 PSM13 was kicked off by a pre-sprint. 10+ of us rolled into Oshkosh early, with the intention of focusing on the portal_skins removal plip. I arrived a day later than the usual suspects to see that things were already kicked off on redefining the javascript integration story for Plone which generally distracted us from the goal of coding for and drove most of us to the overarching question of...  Seriously, can we define Plone 5 for real and really mean it?

Plotting, coffee, arguments, persuasion, a whole lot of momentum, and 2 days later, a document was born. That's right folks, its in writing, and its happening. Get ready.
Alanis Morisette InterludeThis may seem like a very trivial document folks, but I assure you its one of a kind. Not only does it list what we plan to do, but it even has marketing reasons and measures of completeness. Get your lottery ticket now. Book a flight to loch ness and prepare to capture sasquatch. This could never happen again.
I'm sure you don't trust us, and considering that the first talk of Plone 5 like things started in December of 2007, I totally understand. Why is this time different? First, all of these features are already in progress, and in fact many of them are complete. There was talk of a 4.4 release and that is likely hitting the wayside in exchange for the opportunity to make many overdue, much needed improvements (read:change is coming). 
This Time, in the Key of A MajorWhat is a major release? A major release is an opportunity for us as developers to break some backwards compatibility in exchange for overdue large infrastructure upgrades and extra fancy new features. These changes affect highly customized sites the most so if you are running stock Plone, chances are that you are going to be A-OK. In general, we do our best not to break anything for ANY upgrades but some changes can't be avoided. Hence - major release!
Second, we reached out to integrators at the conference, business people, developers, etc to get feedback. What you see documented is by no means final, but rather a general consensus of what generally sounds reasonable to cut a major release. Think something is missing? Submit a PLIP NOW. Not sure why something is or isn't there? Ask a question in the comments and I will gladly explain. The posted document has open comments and if you want to have access to help us gather collective business reasoning, just ask.
Deco vs Collective.Cover or How I learned to Stop Hating and Start WAITINGOh dear. This was very much debated, but you'll notice these are both omitted from Plone 5. If we were to include either one of them, there is a good chance the release would be delayed even more, and both technologies have some serious technical issues before they can really be integrated. In addition, it was crossing the line of "maybe too much at once". They can still both be used as ad ons and I encourage the continuing development of both.
Third, we realized that the basic changes present a need to prepare add-on developers and integrators for the set of changes that are coming up. We need to get the community on best practices NOW so that the shock of Plone 5 doesn't hit us later. Almost all of the technologies and framework to make your custom integrations and add-ons compatible with Plone 5 are available for use TODAY. However, modern best practices are not documented well, and we realize that the success of Plone 5 relies on people upgrading their Plone 2.5 habits to modern development and integration practices. Documenting these best practices is at the top of all of our lists and we will do what we can to make sure that you aren't shocked. 
Tough Love InterludeI know integrators, it's hard to change old habits. But, progress demands it. If you are particularly concerned about a change to the infrastructure and how it will affect your deploys, we will be putting together a FAQ to go with the release. Start asking here or on the PLIPs or on the Plone 5 document. We'll make sure that you're "story" has a translation. We aren't ask you to go back and upgrade all sites, just that you become familiar with modern Plone practices and start off new sites with them. A quick list:
  • Theme with Diazo
  • Use z3c.form for forms (not formlib)
  • Don't make CPT workflows (not like you wanted to anyways)
  • Use BrowserViews not portal_skins. Once you start, the fun won't stop. 
  • Stop making custom archetypes and use Dexterity
  • Think about your reliance on z3c.jbot: in theory that is a task for Diazo these days. UPDATE: People are bitching about this statement. I don't know what to tell you. I prefixed it with "in theory"  and a little "Think about" because I have never seen a need for jbot myself. That's right, I have never used it. I'm sure it's great, but my general thought is that it is a crutch for a more real problem that needs to be solved. This is not part of my expertise or something I care to fight about or elaborate on so here is my advice: do whatever you want. This blog post is not here to prevent you from thinking.
Fourth, we have no BDFL or dominant company at the moment to drive development and set a roadmap (and we are excited about it). Many historical leaders and companies have moved on to different places in life. All of the statistics show the community moving to a less centralized development pattern and at a faster rate (more on that research in a later blog post). When you see the data, you'll get tingles I'm sure. This means that we need to coordinate differently and make decisions differently.

Because of this, we realize the need to reach out for help to hit this vision so here is your ample notice and opportunity for all of the community to help us reach this goal. We aren't the community that can lock Martin Aspeli in a room and expect Deco to pop out a week later anymore. We have to be a bit more deliberate and communicate much more clearly. Someone said "failing to plan is planning to fail" and I almost spit out my beer. Fuck that shit, I'm planning to kick ass.

In particular, we are reaching out to companies to help us support this vision by offering resources to help out a particular PLIP. For example, SixFeetUp is offering QA time for new features like plone.app.events and Netsight is interested in helping get the Plone login story fixed (with one of my clients also having a vested interest). Please check out the story and ask your boss, clients, etc if there is some way that your company, institution, whatever can get behind a change and help is make Plone 5 better. I'll update this post with anyone who pledges to chip in!
We Need Marketers, not DevelopersOne thing that you will hear more of as these blog posts start pouring out of my butt is the notion that we aren't really suffering from lack of developers. We can always use more of course, but what we REALLY need are people with "soft skills". We need people who can start getting ready to market and push, who can make campaigns, and translate developer speak to business leaders and decision makers. Got some of those lying around? Contact me and we'll take advantage of them tomorrow.
In summary, Plone 5 is happening this time because we now understand the importance of communicating with all of you NOW and not after. All of us have promised to blog, talk, and communicate more. In exchange, we are asking for your help. Help as developers, integrators, and most importantly, business people and marketers. Key contacts are listed on the document and hopefully each leader will write a post explaining the change and what it means to you. When in doubt, just ask them.

Which is a perfect segue into my personal vendetta Plone 5 project: removal of portal_skins.

Goodbye CPY

Wildcards fantastic home brew
At the Plone Conference in 2012, I started working on the removal of cpy/vpy scripts. It's been slow and I confess I haven't dedicated the time I wanted. At this moment I think its for the better because there have been some realizations in the meanwhile on the right way to do this.

Initially I was focused on only upgrading cpy/vpy scripts but have broadened the scope to idealism in the way that only a masturbatory developer could: Down with portal_skins!

Now before you freak out, let me say that YOU, integrator, developer, all around good guy can still upgrade your plone product which uses things in portal_skins. However, my vision has a portal_skins free Products.CMFPlone. Everything that is there will be moved into plone_deprecated and you can let acquisition be all magical and do weird stuff still. In addition, your custom skins and what not will still work (there is no guarantee that this will still be the case in Plone NextBigRelease though). However, a major goal of Plone 5 is to return core to represent the current best practice of developing in Plone

And now the most important question: WHY? portal_skins are part of this notion of acquisition. Security and performance reasons have show us that maybe this isn't the best way forward. In addition, modernizing the underlying framework relies on us moving beyond this notion. But wait, there's more. Most of that stuff existed before test cases were culturally enforced. The code is untested and intricate, so people are terrified to touch it. We can't fix bugs on files that everyone is afraid to modify. It get's better. Customization of page templates in portal_skins is shotty at best. It doesn't survive upgrades. Moving all forms to z3c.form gets us on one form technology that can be plugged and extended, and should survive future upgrades much better. Last but not least, times have changed. Resources like js and css shouldn't be managed in the registry. Anything bigger than a small site should be compressing with modern tools and hosting in a web server and/or a CDN. We need to recognize that the industry standards have changed and we are behind. Time to play a little catch up.

Moving all of portal_skins is a much larger task and this meant changing the approach to the change. At first it was a branch on Products.CMFPlone but I had a couple of realizations that led me to believe that the majority of changes should actually be done in a new package, which will then be merged back into core at the end. I'll bullet point the reasons since I'm running out of transition words:
  1. Constantly merging back in changes in every package made me want to throw my laptop out of the window. Git has been merging like a dickhead lately and I had several errors from that waste a lot of time. 
  2. I want these changes for my own projects. Yesterday. I am willing and have the capacity to put them in live sites. Mostly performance reasons, but also integration pain points. I suspect other people do as well.
  3. We can document and keep track of the effort in a more sane matter. When merging in commits I was losing track of the W5H of what happened because this is such a long ranging project.
  4. Integrators and add on developers can pull the changes at any time and see if their shizzle works.
  5. We have a lot of test cases to write. I'm emotionally tied to this thought that we can start fresh and outside of core will just seem more approachable.
All changes will go into plone.future.* packages that will be grouped by general integration points.

Grace Lim, the Plone Ranger
People will be able to grab them at any point. At the moment I see these as:
  • plone.future.formscripts: All of our core actions like a add, move, delete, etc.. are on code that hasn't been touched in YEARS. These are things that are highly unlikely to be uncustomized. This also includes control panel forms and their conversion to z3c forms. I can't imagine that most people can't start using this as soon as a release is cut. At first it will be just a few scripts and I will cut releases after each new thing is added. Ideally we can get these out in the wild, slowly, before the big bang.
  • plone.future.login: Oh god, login is worth a whole blogpost at least this length. It can't be approached in the same set of changes with any sort of sanity. 
  • plone.future.resources: Plone 5 will have a new diazo theme so this won't affect you. Old themes will still work, but if you have properly based your site on Sunburst, then you have the OPTION to use this package, which will contain 1 properly minified js file, 1 properly minified css, and various icons moved out of portal_skins and into resource registries. I know I have several sites that likely won't redesign for a while and will benefit immensely from this. 
In order to smooth this transition and to allow you as an add on developer and/or integrator there are a few other things that I intend to include:
  • A script to scan a directory of code that issues warnings for usages of things that have been deprecated.
  • A script to scan a directory of code that issue warnings for all plone.future references once the packages have all been merged back into core.
  • An intense upgrade guide that matches said warnings and indicates old school vs new school ways of doing things.
  • Tests, tests, tests!
Ideally each one of these items is updated AS the feature is removed to limit the last minute documentation work. I'm in the process of using cut, copy, paste, delete, and rename as an example and setting up this framework, after which I'll start cherry picking work that has already been done and calling for more help. The hard work and dedication that went into plone.api has been inspiring and its my hope to walk away from what could be a tedious task with an immense amount of pride. This is a chance to fix a whole helluva lot of things.

Overall, a pre-sprint was the best thing to happen to a symposium/conference since reliable wifi. 

Fundraising

When you thought this blog post couldn't get any longer, it did. It will continue to grow until you all REGRET  knowing what's going on. Should you have made it to this point, obviously you haven't learned. 

It's like the city of Oshkosh just knew...
Being on the Plone Foundation board has been fruitful this year, and I'm super ecstatic about the push to spend money on sprints and kickstart this Plone 5 process. I was so excited in fact, that I gave a talk about it. With that victory has come a little burden to bear, and that burden is a bear by the name of "fundraising".

In order to dump money into sprints this year, I promised to raise the money we dumped for next year, so we can do the same thing. Turns out that I am a TERRIBLE fundraiser with a serious time deficit and intense lack of skills. Enter Jason Lantz, who gave a talk on how he built a fundraising platform on Plone for the innocence project. How disgustingly appropriate! After spending a full day hashing out ideas with him I realized how little I knew about raising money for non-profits and how much beer I will owe him the next time we meet up.

He has been a community lurker for over 10 years and is very excited to help us, the good people of Plone, get our fundraising shit together. His work is open and he is working on repackaging the add on for use by everyone (if you are a non-profit and you are interested, head over to github to see the progress). I am excited to work with Jason and in addition a whole new team of people excited to kick off fundraising efforts. Expect to hear more from us on that later and remember: you don't have to wait for a fancy donation website to support the foundation. Go to plone.org to help out now. The money that you donate is going to things like bringing K.K. Dhanesh over to Oshkosh from India to sprint for the first time!!

Communications

I've hinted at the severe lack of communication that goes on in Plone and we have lots of people (including myself) that have vowed to make this a little easier on people. This includes but is not limited too:
  • Eric Steele, Rok Garbas, David Glick, and myself vowing to blog more. Twitter has killed the blog and we need to go back to putting complete thoughts and ideas on paper for those that can't explain MultiAdapters in under 140 characters. I encourage all developers to get back to the paragraph.
  • Eric Steele will be trying to get together a regular call of team leaders, which will be attended by Rose Pryune who will act as a translator and post back to the community. 
  • A revamp of the marketing team to be led by Eric Rozeboom. There is a lot to be done here and I'm sure he will post a wonderful plan soon and start recruiting. I know in particular that I am looking for someone to help us better utilize our adwords account. Know someone???
  • Heidi Reinke of UW Oshkosh College of Business has volunteered to have her marketing class for fall semester take on Plone as a use case for marketing. I can't wait to see what she comes up with and if you get contacted out of the blue by people doing research - don't be shocked! 
And look guys, its in writing. No backing out now!

Lots of talk about communication led to some solid realizations on all sides. Developers don't like to communicate in lengthy posts like this and a lot of "translation" gets lost because of it. Forcing people to write docs is, however, not the solution. The oral interview is MUCH more effective and we have to spread that word. Want to know what's going on in Plone? Don't wait for a blog post. Ping a core member, ask them on a call, and write up a post for everyone else to consume. 

That Stinking Contributor Agreement

In addition to being a Plone Foundation Sponsor, hosting a kick ass pre-sprint barbecue complete with home brew, and maintaining several high quality Plone plugins,  Talin Senner of Wildcard Corp has volunteered to help us finally kick this long standing issue with the Plone Contributor Agreement. I am eternally grateful for them picking this up and rolling with it since it was making my head spin and I was dangerously spread thin.

Mailing Lists

We REALLY need someone to put a nice front end on mailing lists. The lovely Heidi made a statement along the lines of "why would i use plone? signing up for the mailing list makes this software look like its from 1990". OUCH. Interested? Ping me. We just need a plone.org initial UI into the basic mailing lists. 

In addition, we should think about making plone-accounce a mail chimp list, considering that:

  1. The UI doesn't suck at all
  2. It's for communicating out, not within

I'm Running out of Steam

I didn't even get to talk about Grace Lims video project for coders and their kids, the Plone Ranger, the packages that were released and the intense code sessions, the work on EVERYTHING from coredev docks to mock, realization that we need help with adwords... oh god. That's another 6 hour plane ride of info. For now, sleep.

More sprint reports at:
Followup Awesomeness: