Wednesday, November 17, 2010

Plone, JQuery and Prototype: Easier Done Than Said

It's hard to believe, but jQuery hasn't always existed. So some of us used other libraries, like Prototype[.js]. We even built entire systems on them. And then upgrade day came...

I sat back with my largest bottle of beer and got ready for what was sure to be a javascript apocalypse. How could I merge 4 years of Prototype based javascript with Plones new JQuery libs? Surely it would be painful.

And yet it wasn't. It will take me longer to write this blog post. But just in case there is some other straggling soul like myself who just can't get enough Event.observe(), fear not! Follow these simple steps:

  1. Create a new js file and add one line: jQuery.noConflict();
  2. Then register that js file with your site. Prototype must go first or it will whine (remember not to compress or cook!). The noConflict js file must be listed before using any custom scripts. I recommend loading it right after jQuery.js. See below for a sample.
  3. Reinstall. 

Much can be said about jQuery playing nice and especially to the Plone javascript gurus who were kind enough to work within a namespace. Thank you. I owe you a beer.

Tuesday, September 14, 2010

LA Theme Sprint Report

Kudos to Michael Miller and Luke Brannon of the LA Plone Meetup for hosting the first (annual?) Plone Theming Sprint in LA. So many things about it were successful that I have had major writers block on how to get things started so I'm just going to jump write in.

The most exciting part of the sprint was not only the diversity of the people, but the diversity of experience as well. Plenty of newbies overcame their fears to tackle new skills. For example, Heather released her very first product to pypi, almost the whole crew picked up and ran with XDV, and Alice was an end user who just showed up and by the end of the first day was familiar with installing products through buildout, basic site admin, and fixing borked databases!

Why would her database get borked you ask? Because she went through and installed all 52 themes listed on plone.org to evaluate quality, take screenshots where necessary, and took copious notes of all posted themes that didn't uninstall cleanly or didn't install at all.

Now you are asking, why would someone do that? Only because we tackled the revamping PloneSoftwareCenter! This included adding ratings and we needed some data for when it goes live. Oh yeah. Alec took plone.contentratings from bad ass to bad asser while I worked on integrating and revamping the UI. Check out the screencast to get a preview of the changes. This will be a great accompaniment to the Plone 4 release so if you have any time at all to help get this deployed - ping me!

And finally you ask, why would you want to revamp PSC? Glad you asked. Plone.org has less than spectacular support for navigating themes and we needed to have a beautiful place to display all of the new themes that the rest of the crew came up with. 3 Mikes, Trish, Tyler, and Albert banged away on creating 3 beautiful XDV themes (coming your way soon after solving the packaging issue).

The best part of all? At the end all anyone wanted to know was when the next sprint would be. That's a trend I'm happy to obligue. Thanks to everyone who showed up and good work everyone!


Saturday, September 4, 2010

Feed a QA Sprinter

The 2010 Plone Conference is just around the corner and sprint planning is well under way. One of the sprints will consist of creating Selenium tests and integrating with Hudson. As Plone moves towards more and more Javascript and Ajax interface these tests will be essential for maintaining a high quality code base. This sprint will be great for newbies and veterans alike with some potentially boring but definitely worthwhile QA work.

Ideally this sprint is crowded and short so that people can get back to their regularly scheduled sprints. If you will be in Bristol for the sprints, please consider donating 1-2 hours of time to help us get up and running. If you won't be in Bristol and want to remote sprint stay tuned! However, if you can't do either (or just hate doing that stuff) but want to support the effort, please consider chipping in to feed a sprinter.

If anyone has other ideas on how to lure people in or has items to donate as door prizes/giveaways/whatever I'm all ears. Additionally, if you have a company and/or want to sponsor this sprint directly let me know and I'll cancel the chip in.

Happy bug hunting!

Wednesday, July 21, 2010

Changing the Order of Fields in Archetypes

I wish I knew this 4 years ago.

When creating new content based on archetypes, I usually start by copying the base schema, or one of my own base schemas with something like MyNewSchema = BaseSchema.copy(). One thing that has been historically hard in my case is that the schema order in the code directly correlates to the order in which its displayed. This means that the base copied data always goes first when displayed, which was not always what I wanted. The solution was custom templates and blech all over.

Apparently you can just reorder these things. Deep in the bowels of archetypes there is a function called moveField. Now, reordering the schema can me as easy as MyNewSchema.moveField('description', pos='bottom').  Hot dog! Pasting the interface for reference.

Friday, July 16, 2010

The Culture of Reporting Bugs

Did you know that Plone has a QA team? It does! The QA team is just getting off the ground, so it's a great time to talk about the culture of quality assurance. Since I don't use enough old english in my posts, I want to kick-off the topic with a couple of QA Commandments that when recognized by all members of the community ensure a yummy culture of quality. The first part involves bug reporting, and they go something like this:


1.Thou shalt report every bug, big and small, content, documentation, infrastructure, and beyond
Is it a spelling mistake? Report it. Is the interface about as useful as a bag of sand? Report it. Is it impossible to find the documentation you are looking for? Report it!

Will your bug get fixed in a timely manner? Probably not. But that's ok. The point is that you noticed something that was below perfection, and you cared enough to document it by writing a bug report.

I worked on a QA team right out of college and I am proud to say we worked just as hard (if not more!) as the developers. There was constant arguing between the teams but together we delivered a super stable end product. A key component of our strategy was competition within the QA team itself over who could report the most bugs. There was no prize, just pride. And it worked! Every nook and cranny of that system that wasn't perfect was documented.

2. Thou shalt not bitch about bug reports
This follows from commandment 1. Good QA people care even more than the end user about a perfect experience, and caring takes a lot of effort. The thing that happens when developers stop complaining about bug reports is that it actually encourages people to report more bugs - imagine that!

A good set of bug reports should be treated like problem documentation. The better documented the faults are the easier it is to fix and the easier it is to provide support until that fix goes through (since it may be a while). 

The best part is that it's easy to encourage this culture. The next time you close a bug, big or small, thank the original reporter. 

3. Thou shalt make it easy for everyone to report bugs 
Yes you! If you want to find lots of bugs you need to make it easy to report them. This makes me nuts with just about every software product known to man. One click to report a bug. No screens, no searching, just get the info and get them outta there. And yes, it's on my list of things to change with Plone.org.

Unfortunately, I think Plone unnecessarily suffers from the bug collecting of zope and zodb. Searching for a zope bug collector without the exact keywords in google returns two bug collectors, one from 2003 and one from 2005. Both abandoned. And if you report in the wrong one they just close the ticket and that's it. Screw you end user for not being able to differentiate between a plone, zope, or zodb bug. Boo.

So I am putting a list of collectors here for my reference as well as yours. Go ahead, report that bug you've been sitting on for a while. 
Now, if a bug gets marked invalid because it's someone else's problem you can just move on to the next tracker and follow through there. Did I miss any? 

4. Thou shalt stand by that bug and not be bullied by developers
This is the hardest part of any QA effort. It's easy to give up when a developer spouts some mojo that you just don't understand. Nonsense poopy pants! Good QA people are so annoying developers wish they would go bury themselves in a hole in a forrest. And you know what? They are often wrong. But a lot of the time they are right, and a HUGE bug manifests from what seems like a template error. That's when it pays off to be a PIA.

So stick to your guns QA jedi's and do what's right for the end user. If you always keep them in mind the end result will always be what's best for the product and the community.

Thursday, June 10, 2010

Diversity in IRC

Confession: I was a closet female for a long time. My nick is purposefully ambiguous and when people assumed I was 'Eric' I never corrected them. I never signed my full name, not because I was afraid or insecure in my answer, but because I was afraid I would get treated differently for it. None of my profiles had my picture until just a few months ago, when I "came out" at the Plone Budapest conference.

I usually feel bad about it. For 7 years I could have been supporting women in technology. It seemed the easiest way to get by at the time. Was I being a coward? Maybe I was making it all up?

Then there are days like today, where a I get a gentle reminder as to where it all started. To the left is a screenshot of an #plone IRC chat with someone who was just completely lost in their task (probably still is). Many of us were helping out this person and at some point they moved it to a private conversation. Fine - that happens a lot. 

Click the image to get a "x-large" more readable version. I'll wait...

I'm not posting this to complain* - in fact I see it as a victory. I have no hard feelings and a simple correction not only took us back to business but raised the standard for getting help from that point forward.

Rather, I want people to understand why it's important for groups like PloneChix to exist. Diversity is scarce in the community and dealing with it is still new to many people. And guess what? The only way to make things better is to be more visible and pull through awkward moments like this. 

It's easy to dismiss this as "not a Plone problem" when it's plone "users" that are the offenders but that is a cop out. If someone is new to helping out on IRC and is treated like this, we risk permanently losing their help and expertise. If it's normal to have women in IRC, at conferences, etc... then the standard for behavior will be higher to begin with.

So ladies, help me out - I'm tired of dealing with this alone. Take a moment to be seen: get in IRC, go to a conference, talk at a local meetup and invite your friends. One at a time, we can make Plone a stress free place for everyone.

*I spend way too much time arguing with people about whether or not this stuff still exists. Many think that it's a problem of the past because it doesn't happen in public forums or chats anymore. My response is usually "Duh! Of course it doesn't." Most incidents DO go unreported for that exact reason.

Tuesday, June 8, 2010

Searching the Plone3 API

After kvetching about the Plone API documentation, I realized that maybe it's not the docs that need updated, but rather just a better way to navigate through them. Inspired by an awesome article by dukebody, I looked into how chrome could better serve me. To set up API search in the url bar:

  1. Chrome > Preferences > Default Search: Manage
  2. Click the + sign
  3. Choose your name and keyword. I used 'Plone 3 API' and 'api'. Use this beast for the url: http://www.google.com/search?hl=en&q=site:http://api.plone.org/Plone/3.0/private/products/+%s&aq=f&aqi=&aql=&oq=&gs_rfai=
  4. Save and enjoy.
In my case to use it I just enter 'api PloneModuleName' into the url bar and click enter. This gives me instant gratification and filters out the weeds of versions I don't want. Plone on.

Tuesday, May 18, 2010

Adventures in Buildout: Zope 2.12 meets SOAP

After 3 years of hanging around in Plone 2.5, I am finally getting around to upgrading our infrastructure to Plone 4. Part of that upgrade includes pulling out all of our services and api calls into a separate buildout, a solution that does not actually need the Plone package. A big problem with this migration was that we still use SOAP to talk to external vendors, most python SOAP modules suck, and none of them are fully integrated with a database/framework.

As a satisfied user of SOAPSupport for many years, my first inclination was just to continue with an old version of Zope and continue to use that.  However, the project hasn't been supported/updated in a long time and I was excited to take advantage of memory improvements in Python 2.6, a fully eggified Zope, and Blob storage among other things. There is a great article on configuring SOAP in Plone, which targets Zope 2.11 and custom products, however its still very mind bending for those of us with simpler needs.

While updating the z3c.soap package for Zope 2.12 compatability, I was also busy trying to grok "new" buildout concepts (I know - I've been in the closet for a while). I couldn't find one really solid simple tutorial on getting started with buildout. After a roller coaster ride of emotions and awesome help from the Plone community, I *think* I've figured out the simplest way to get any python SOAP server up and running with (or without) buildout. I have not addressed the WSDL or complex types questions yet, since I don't need them. I also said simplest, which means I had little regard for proper classing and blah blah blah... yell at me in the comments if it's grossly off-key or if I missed something - it's bloody likely.

Below is an introduction to configuring SOAP in Zope as well as an extra detailed buildout process for old-schoolers like myself who are used to Products based install. I hope it helps someone get up and running with a python soap server fast, in a repeatable, buildout style.

FYI, this can also easily be customized to support Plone and other zope based apps since its buildout based.  My goal was to make this like MySite, where you can start and build off of it. In the end I don't think I have even come close the greatness that was MySite but you can still download the finished package if you don't care about the how and just want the now, or if you want to follow along easier.

NOTE: At the time of writing virtualenv has a bug with this buildout so if you get an ImportError for shutil, just use regular old python. This is also why this tutorial is not using virtualenv, although its perfect for this scenario otherwise.

Summary
For those who don't care about buildout or the details, here is the recipe for simple zope + soap. Customize away!

The Details: Creating the Buildout
I won't go into the philosophy of buildout since its all over so we can get to implementing. I keep all of my buildouts in a directory in my home folder, called 'buildouts', so I always know where to start. Also, before you get to the end and kick yourself, I'm using Python 2.6 here. I have not tested it with any other version. Caveat Emptor.

First, make sure that you have the buildout package installed:

> sudo easy_install zc.buildout

Create a buildout environment, with all the scripts we need to get started
# create a new directory for our new buildout
> cd ~/buildouts
> mkdir zsoap
> cd zsoap
# initialize the buildout environment
> buildout init
Creating '/Users/eleddy/buildouts/zsoap/buildout.cfg'.
Creating directory '/Users/eleddy/buildouts/zsoap/bin'.
Creating directory '/Users/eleddy/buildouts/zsoap/parts'.
Creating directory '/Users/eleddy/buildouts/zsoap/eggs'.
Creating directory '/Users/eleddy/buildouts/zsoap/develop-eggs'.
Generated script '/Users/eleddy/buildouts/zsoap/bin/buildout'.
# optional: include bootstrap.py
> wget http://svn.zope.org/*checkout*/zc.buildout/trunk/bootstrap/bootstrap.py

You don't have to include the bootstrap.py file, but it makes things hella easier and I'll assume that you have it for the rest of this. After initializing the buildout, you will have a bunch of extra directories in your zsoap folder, including bin, buildout.cfg, develop-eggs, eggs, and parts. Don't worry about them for now - just make sure they are there.

Next, edit buildout.cfg so it looks like this. I'll explain it all while we wait for the buildout to run.

Finally run the buildout:
> python bootstrap.py
> ./bin/buildout

Note that whatever version of python you run bootstrap with is the version that zope will be running with. This recipe was created in Python 2.6, since Zope 2.12 runs swimmingly on it. If your system python is not 2.6, make sure to alter your bootstrap command to reflect that. 

While its running, you may see errors like "SyntaxError: ("'return' outside function",...". Don't worry about those. It's simply trying to compile scripts, which you can't do.

The Breakdown
Once the buildout starts going, go ahead and take a shower, walk the dog, or read the line by line description of what is happening:

1. [buildout]
2. parts = scripts
3.             instance
4.             test
5. 
6. extends = http://svn.zope.org/*checkout*/Zope/tags/2.12.5/versions.cfg
7. versions = versions
8. eggs = z3c.soap
9. extensions = buildout.threatlevel

Lines 1-4 say that this buildout has three parts that need to be run. When you run this, you will see that the "parts" directory that was automagically created from initializing the environment now has 3 folders with the exact same names. Coincidence? You decide. 

Line 6 is a link to download an automatically configured buildout for installing zope 2 in a delicious eggy format. To change versions of zope, just modify this url. Go ahead. Try it.

Line 7 Indicates that we are going to force some packages to get a certain version, instead of the latest. This is called "pinning" a version. Some packages have conflicting versions, and you may need to pin the version which satisfies everything. This line simply tells buildout that there is a section, called versions, which will do just that.

Line 8 indicates which additional eggs we need to get the project going. It will ask PyPi for the latest version of this eggs (unless we pinned it) and install them. Notice that we didn't have to include Zope2 because it's handled with the scripts section with a special recipe that you'll see later.

10. [versions]
11. Zope2 = 2.12.5
12.    

10-11 show how to pin a version of a package. This is not necessary for this case since the Zope2 recipe does this for us, but other packages probably won't have this luxury. Maybe you want to pin to a minor version, for example. Furthermore I already numbered these lines and I'm too lazy to go through and renumber them.

13. [scripts]
14. recipe = zc.recipe.egg:scripts
15. eggs = Zope2

Lines 13-15 might as well be magic. They install Zope2, the inner workings of which I have no idea.

16. [instance]
17. recipe = plone.recipe.zope2instance
18. user = admin:admin
19. http-address = 8081
20. products = ${buildout:directory}/products
21. debug-mode=on
22. zcml = z3c.soap
23. eggs = ${buildout:eggs}

Lines 16-23 create a new zope2 instance. It's pretty self-explanatory except for a couple lines. Line 22 males sure that we include the z3c.soap package, and run the zcml slug that initializes it. This patches the publisher to make it accept soap requests. Line 23 makes sure that all the eggs created in other places of this buildout are put in the path of the instance. Without this, it won't be able to import anything.

24. [test]
25. recipe = zc.recipe.testrunner
26. eggs = z3c.soap

And finally, we must not forget to include test cases to make sure everything is running smashingly. 

Up and Running
Assuming everything runs correctly, we can start testing things out. First we want to validate that everything looks like its running ok.

> ./bin/test

It's that easy. This should run fine and pass all tests. Yeehaw! Let's start the new zope 2 instance in foreground mode:

> ./bin/instance fg

Still easy. Woohoo! If the slug was installed correctly, z3c.soap will have a line in the trackback that says something like "INFO Zope z3c.soap: modified ZPublisher.HTTPRequest.processInputs". Congratulations, you are ready to serve SOAP requests.

Taking it a Step Awesomer

Now what? Well, you can follow the original tutorial and add wsdls and custom type and all that fancy stuff. Since I'm lazy I prefer to do everything through scripts. In the ZMI, I like to add a folder called 'services' and then just pile up python scripts in there. Err... I mean Script (Python)'s. Let's verify it's all SOAPed up like we want. Add a script called 'test' in a folder called 'scripts' that simply returns a string 'hello world!'. Then you can use the SOAPpy package to see what's really going on. From the python prompt:

>>> from SOAPpy import SOAPProxy, URLopener
  >>> url = "http://localhost:8081/services"
  >>> namespace = "http://plone.org"
  >>> server = SOAPProxy(url)
  >>> server.config.dumpSOAPOut = 1
  >>> server.config.dumpSOAPIn = 1
  >>> print server._ns(namespace).test()[0]
  hello world!

Every script you add will have a SOAP interface built in. How cool is that? Also, a tip from the trenches, use dictionaries to pass back values for more complex values. In my experience, it's more compatible with Java stacks, which I assume you have to deal with, because otherwise you wouldn't be using SOAP. I digress...

So I know you're thinking, what is the awesomer part of "taking it a step awesomer"? Well, if you want a repeatable buildout you don't want to store your scripts in the ZMI. I know I didn't, since we have an SVN repository that is way better for such things (again, I'm behind, I'm not cool enough to Git or Hg yet). We can do is use the CMF Filesystem Directory view and store all of our scripts on the filesystem. While they are not editable in the ZMI, but the can be executed and customized.

I tried my hardest to find a way to do this without making a product, but I can't find a way to register a CMF directory view without being in the context of a product. So let's walk through making one - you probably need it anyways. It's slightly annoying, but at least this way it won't get wiped by re-running buildout. If you already have a product then skip the paster part and just add the necessary slugs.

First make sure that you have access to paster. If you can't use paster from the command line,
> sudo easy_install -U ZopeSkel

Then decide a name for the product, and run paster. I will use z.soap for this purpose.
> cd develop-eggs
> paster create -t basic_zope
> ...blah blah blah

Next we need to add our new egg to the buildout.cfg, mark it as a development package, and include it in our zope instance. Update your buildout.cfg to look like this, with the bold lines indicating the changes. Hopefully by now the buildout thing is starting to make sense. But we aren't done yet.

Now we need to actually register a services directory. Let's make the directories first
> mkdir ~/buildouts/zsoap/develop-eggs/z.soap/z/soap/skins
> mkdir ~/buildouts/zsoap/develop-eggs/z.soap/z/soap/skins/services

All of the Script (Python)'s that we want to be version controlled can go in the services directory. You can version control the whole buildout, or of course register your new egg with PyPi and replace that whole mess with eggs = my.egg.

Tell zope about the directory that you created by editing your new eggs configure.zcml, which will be in ~buildouts/zsoap/develop-eggs/z.soap/z/soap/configure.zcml. Add the registerDirectory directive to point to the folder you just created, adding the cmf namespace if needed. The final configure.zcml should look like this, with the changes highlighted.

Phew! Are we done yet? No. Re-run buildout and restart your instance

> ./bin/buildout
> ./bin/instance fg

At the root of the ZMI, add a Filesystem Directory View and you will see that your directory is listed. Let's put the same test script that was in the zodb the first time in the filesystem now. In your new skins, services directory, add a file, test.py with the contents below and run the SOAPpy test above. Viola! Now you can add scripts until you are sick. They can compute, forward info, act as an API, and even interact with your products. No restart needed to add/change scripts since its a filesystem view.

##Script (Python) "test"
##bind container=container
##bind context=context
##bind namespace=
##bind script=script
##bind subpath=traverse_subpath
##parameters=
##title=
##
return "hello world!"

But wait, you don't want to do that last step manually? You want that automated? I did too! Last but not least, let's add the directory view on startup if it's not already there. There is a special initialize function that we can call on startup. Open up develop-eggs/z.soap/z/soap/__init__.py and make it look like this. Now instead of calling zope2 initialization, we need to call our own package initializer by modifying the configure.zcml to look like this. Now start up the zope instance and it will come alive!

So, if there is anything wrong here or it could be done easier, please comment and I will update. The last part was definitely not as easy to setup as I would have hoped. And because I'm so nice, you can download this package, untar, bootstrap, build, and go crazy with customizing. 

Monday, February 22, 2010

Catalog Key Errors

Apparently I am the patron saint of key errors. The latest one was with the catalog, and it was way nicer than some of the previous monsters because at least it didn't bring down the whole site or involve sweaty, hand shaking stress. To the end user, certain keyword searches didn't in livesearch, advanced queries. This key error manifested in the logs as:

Module Products.ZCTextIndex.BaseIndex, line 203, in search_phrase
  Module Products.ZCTextIndex.OkapiIndex, line 161, in _search_wids
KeyError: -1525983394

I had a feeling that this would happen because a few days earlier, I came across a few weird errors trying to emply some jedi mind tricks in the db. An example of one:

2010-02-17 21:46:36,505 ERROR @/opt/Zope-2.9/lib/python/Products/PluginIndexes/common/UnIndex.py/UnIndex.py UnIndex 194  : DateIndex: unindex_object could not remove documentId -1324564720 from index modified.  This should not happen.
Traceback (most recent call last):
 File "/opt/Zope-2.9/lib/python/Products/PluginIndexes/common/UnIndex.py", line 168, in removeForwardIndexEntry
   indexRow.remove(documentId)
KeyError: -1825924534

Whoops.  They didn't cause anything obvious to happen so I did what every lazy sys admin does and ignored it. As far as I can tell there isn't a required correlation between the two errors, but it will at least point you in the right direction.

That was Friday night. Then customers got on the system Monday and we started getting the ZCTextIndex KeyError sporadically. It affected all searches with the word 'jackson' and using indexes set up with OkapiIndex algorithm (PloneLexicon). This is the default choice for the Title, SearchableText, and Description indexes.

Looking this error up on Google shows a few answers, which involve clearing and rebuilding the catalog. I'm sure this works, but our index alone is 1.2GB so that was really a last resort option. It would take so long that before reindexing we probably just would have monkeypatched the catalog code to just ignore those keys. I was determined to find a way to fix without rebuilding. And I did.

Warning: the following explanation is based on eye crossing interpretations of zope code. I may be off a little but I think its a worthy attempt nonethless.

The text indexes need a way to correlate a word set (what the user is searching for) with a document and vice-versa.  For example, when I lookup 'jackson', I need to know that 17 documents have that keyword. If you have been mucking around with the catalogs or something went crazy on your catalog (like your sys admin) then you can get this words list of sync with other indexes and the real catalog. To confirm this, try idx.getEntryForObject(docid) on the failing index, where docid is the KeyError integer in your error log. In my example, the keyword 'jackson' in the words BTree pointed to a list of documents, all of which were valid except -1525983394. By just extracting the bad reference to this non-existing document, then there is no need to reindex everything.

Note that for this to work you need to know the word that triggers the error. You can't do a reverse lookup to find out which words reference that document because that mapping is the one that is still correct. It will always return empty due to a silent fail. If your users won't tell you, then you can just log it when it errors out later. If its really bad, iterate through all words and what they point to and see if they throw a key error (hint: use the _wordinfo.iterkeys() iterator + other tools listed here). Or just reindex.

As always, backup before hand and if you are nervous work on a copy first. I'm sure this can easily be modified for other indexes/schemes as well.

From the debug prompt:
'''
@docids is a list of the key error items as ints, not strings
@word is the word that triggers the index error
@cat is the catalog object that has the error, i.e. app.foo.portal_catalog
'''
def removeFromWordCatalog(docids, word, cat):
    import transaction
    for docid in docids:
        for indexObj in cat.getIndexObjects():
            try:
                itype = indexObj.getIndexType()
            except AttributeError:
                continue # some indexes don't implement this method
            lex = indexObj.getLexicon()
            if itype == 'Okapi BM25 Rank':
                wids = lex.termToWordIds(word)
                idx = indexObj.index
                for wid in wids:                
                    try:
                        # using excepts because BTrees key lookup is annoying
                        # see if its even really an error before ousting
                        blah = idx._wordinfo[wid][docid]
                    except KeyError: # not listed - nothing wrong
                        continue
                        
                    idx._wordinfo[wid].pop(docid)
                    idx._wordinfo[wid] # persistance
                    try: 
                        idx._wordinfo[wid][docid]
                    except KeyError:
                        print "sucessfully removed"
                    transaction.savepoint(1)
                    transaction.commit() 

docids = [-1525983376, -3423423445, ...]
word = 'jackson'
cat = app.foo.portal_catalog
removeFromWordCatalog(docids, word, cat) 

UPDATE: I already got to the point where the word by word thing became annoying so I wrote a generic function to "unword" a catalog entry. Much nicer, and although I thought it would take a long time it was actually not toooooo bad. I also pieced out the code so I could get as generic or specific as I want in the case that something else happens. Which it will. Woohoo!

def removeDocFromWordlist(idx, wid, docid):
    import transaction
    try:
        # using excepts because BTrees key lookup is annoying
        # see if its even really an error before ousting
        blah = idx._wordinfo[wid][docid]
    except KeyError: # not listed - nothing wrong
        return 
        
    idx._wordinfo[wid].pop(docid)
    idx._wordinfo[wid] # persistance
    try: 
        idx._wordinfo[wid][docid]
    except KeyError:
        print "sucessfully removed"
    transaction.savepoint(1)
    transaction.commit() 
    
    
def cleanUpBadDocId(docid, idx):
    iterati = idx._wordinfo.iterkeys()
    while True:
        try:
            wid = iterati.next()
        except StopIteration:
            break
        
        if idx._wordinfo[wid].has_key(docid):
            print "removing %s from word list %s"%(docid, wid)
            removeDocFromWordlist(idx, wid, docid)
            
            
def removeFromWordCatalogs(docids, cat):
    for docid in docids:
        for indexObj in cat.getIndexObjects():
            try:
                itype = indexObj.getIndexType()
            except AttributeError:
                continue # some indexes don't implement this method
            lex = indexObj.getLexicon()
            if itype == 'Okapi BM25 Rank':
                idx = indexObj.index            
                cleanUpBadDocId(docid, idx)


docids = [-1525983376, -3423423445, ...]
cat = app.foo.portal_catalog
removeFromWordCatalogs(docids, cat)

Tuesday, February 16, 2010

PloneChix: Hot and Not

Instead of boring you with a lengthy explanation of our official purpose, I'd like to address what makes PloneChix exciting and unique and then tackle some misconceptions about what people think PloneChix represents. As a followup post, I'd like to invite all PloneChix members to explain why PloneChix exists for them. There will be more meaning and inspiration from their stories than reading an tedious political statement.


What is so hot about PloneChix? PloneChix is ...
  • ... a no flame zone. That means no RTFMs, no stupid questions, no negative feedback, and no smack talk. This is the most important aspect of who we are. Women in technology, especially open source, are more likely to withdraw from a community because of actions/comments/answers perceived as harsh.   By making a special effort to make a no flame zone, we are encouraging people to ask for help and giving them helpful, thorough, honest answers by any means possible. Who knows, maybe all this kindness will be contagious. 
  • ... a place to be heard. Speak up, let us help you, stick around Plone for a little bit longer and we think you'll like what you see.
  • ... a place to ask non-technical questions (although we like technical too). For example: What is the job situation in the bay area for Plone integrators? Does company X have a supportive female developer community? What can I do to market myself better as an integrator/developer/documenter? 
  • ... an open and supportive learning environment. This is the perfect arena to tell a story and get honest feedback. Did you blow that interview because you underrepresented yourself or did you really not have enough experience? Let's talk about it. It's a place to get advice from women with different experiences and backgrounds.
  • ... an opportunity to tackle that project with a group instead of doing it on your own. We have just started working on defining our projects for this year. Why not add yours? If you have wanted to commit to the core for a while but just didn't want to tackle such a huge project on your own or want some help, we'll make sure to hook you up with a veteran who will show you the ropes.
  • ... a place to network. PloneChix is here to share and encourage each other to write that first book or give a talk at that conference. In my opinion, visibility is the #1 issue with women in the Plone community. Be visible, be heard, and get your name out there. There are lots of special opportunities for women in technology, and if we don't take advantage of them women in other communities will (i.e. Ruby women are fierce). Posting job opportunities, calls for papers/talks, scholarships, et al is highly encouraged.
Hot indeed. There are already some strange rumors about PloneChix that have surfaced that I'd like to take a moment to address. PloneChix is NOT...
  • ... a group of man-hating femi-nazis. In fact it would be way easier to justify if we were. However, we do have a tendency to focus on the needs of women and womens issues in the Plone, python, and Open Source Software (OSS) communities.
  • ... a derogatory name. It's an homage to LinuxChix, one of the finest female-friendly OSS movements to this day. There are many other groups that pay tribute to the original, including DevChix, DrupalChix and CodeChix, just to name a few. The word "Chix" has become synonymous with female fronted groups that discourage friendly fire (no RTFMs) and encourage low barrier to entry.
  • ... a "women only" club. Anyone is welcome as long as they are supportive of the cause and contribute positively back to it and/or the Plone community.
  • ... a support group for complaining or flaming, especially about other members of the community. PloneChix is a place for positive feedback, encouragement, and help. We are about positive change, not negative reinforcement.
  • ... looking for 50:50 male to female ratio just for the sake of having a 50:50 male to female ratio. On the contrary, we are all interested in seeing the entire Plone community grow and thrive in the way that Kirrily Roberts* describes.
  • ... a replacement for the mailing list, irc, or other more formal means of communication about Plone. It is an additional communication channel lead by like-minded individuals. It is also a place to talk about careers, meetups, projects, or even just to brag about a new project in a friendly, open environment. 
This is the part of the post where I start practicing what I've been preaching. Girls, boys, aliens, machines: how does this make you feel? Hint: I was shooting for all warm and tingly inside.

If you are interested in participating, male or female, check out our wiki, which has all the information you need to get started.

* Please read this post 50 times. Take a nap. Then 50 more. Then we'll talk.