Bob Voorneveld: Implement Gmail api in our CRM system

See the PyGrunn website for more info about this one-day Python conference in Groningen, The Netherlands.

I am working for Spindle. We are primarily working on a CRM system. Most Spindle programmers used to work at Voys. We are growing bigger and are still hiring, so come and join us.

Why we started building yet another CRM: communication should be the central point. We are building 'HelloLily'. Lily should be funny, nerdy, smart. HelloLily focuses on accounts, contacts, cases, deals, email, phone calls (future).

Focusing on the email part in this talk. Last year I said: instead of the really old IMAP, let's switch to the gmail api.

Implementation currently: Heroku, Python 2.7, Django 1.7, Django Rest, Django Pipeline, Celery, IronMQ, PostgreSQL, ElasticSearch, Angular 1.3, Logentries.

Celery: 1 scheduler, every five minutes sync every account. Two functions to sync email: first sync and incremental sync. Many functions for sending, moving, deleting, drafting email. This is asynchronous to keep quick responses.

With IMAP we previously had email sync problems: authentication (we should not store your password), not easy to keep track of changes of what happened to a mailbox, no partial download of only one attachment, IMAP implementation differs. Also, searching in the database was not very efficient, even with indexes. We had PostgreSQL problems, models spread over many tables, searching was slow because of that, with every email the search time increased, partial matching was difficult.

So we wanted a search index and use gmail api.

Gmail api: easy api, installable with pip, keeps track of messages (like: since last sync we have 5 new mails and these three have been deleted and one edited), partial download. Since February there is tracking of the type of change. Downside: it is limited to Gmail / Google Apps for business.

Using PostgreSQL with ElasticSearch. Emails are mapped in documents in ES, models pushed to ES with a post_save_signal in Django. Fast response time, averaging 50 milliseconds.

Problems that we encountered and fixes. Encoding and decoding messages, do not trust that the claimed encoding is correct. Sending, forwarding email, with attachments. Losing labels entirely instead of just for one message due to a coding error. High memory usage, scaled up for now. Sending big messages, needed to send in chunks.

Some colleagues are now saying: it does not work properly, let's switch back to IMAP. But we are getting there.

Still beta, testing it out ourselves. See source code on https://github.com/HelloLily

Me on Twitter: ijspaleisje