Tag Archives: bayarea

How IFTTT, Mandrill, and Twilio is helping me find an apartment in SF

Note 7/8/2013: Craigslist makes it impossible (please let me know if you were able to get around it) to allow Mechanize to retrieve HTML from Heroku’s server. However, it works when I send a request from my local server. If anyone knows why it would make a difference, please leave a comment!


Anyone familiar with the SF apartment market understands the tremendous pain and frustrations in finding an available lease at a reasonable price. We have all heard and shared war stories of the hourly refreshing of Craigslist and bringing all of your bank statements and necessary paperwork to the open house just to win a chance to secure a lease for an apartment. In true SF-technology fashion, I decided to minimize response time to a new choice apartment listings on Craigslist by wiring together IFTTT, Mandrill, and Twilio to auto respond with an open email and a click-to-call to my phone number and the number in the listing to schedule an appointment.


IFTTT (‘If this, then that’) allows you to create ‘recipes’ using the simple conditional structure ‘If this, then that’ and various ‘ingredients‘ (i.e. craigslist queries, instagram posts, buzzfeed articles, tweets). It is powerful due to its simplicity and flexibility; it can enable anyone to put the web to work without any programming knowledge.

I used IFTTT to send an email to my email address when there is a new Craigslist query that matches the me and my roommates’ apartment listing preferences. This way, any new Craigslist listing that fits the query that I have set will be sent directly to my email inbox.


I setup a subdomain (I added ‘to’ as the MX record and have it setup on my Mandrill account) responsible to receiving email messages and use Mandrill to parse the inbound email (similar to how I built the A-List inbound parser). In the below code sample, it is important to note that you must first return a ‘200’ to let Mandrill know that it is the right address to send the POST request of the inbound email.

Once the email is successfully received and parsed, I use the Mechanize library to navigate to the Craigslist listing (line 31). Then, a few regex commands to find and scrape an email address (line 34) and a phone number (line 35). With the email address, I use ActionMailer (line 46) to send out a templatized email, cc’ing my roommates, asking to schedule an appointment to check out the apartment.


If I was successfully able to scrape a phone number from the Craigslist listing, then Twilio will automatically make two outbound phone calls (one to my mobile phone and one to the number on the listing). Throughout the day, I will be able to pick up my phone at my leisure and speak with somebody (if the call connects on the other end) to schedule an appointment.

Note that the routing must also be setup in your Rails app (the sample code is not shown here).

The first step is that we are sending a request to Twilio to initiate a phone call to my phone number (agent_number or ‘555-555-5555’ in the example below) with dynamic TwiML (Twilio’s XML-like proprietary markup language).

The first dynamic TwiML (line 38 above and ‘clicktocallcallscreen.xml.builder’ below) tells Twilio to listen, upon a call connecting with my phone (agent_number or ‘555-555-5555’), for a keypad response indicating that I want to be connected with the Craigslist lister’s phone number. If I hit a key, then proceed to dial the other number and connect us. Note that I am  providing yet another dynamic action URL (‘@post_to’) that will populate the final TwiML with the correct phone numbers that will be connected in the final call.

It is in line 46 in inbox_controller.rb where the clicktocall.xml.builder (below) is called dynamically (with ‘@calling_to’ and ‘@caller_id’ both being passed to the TwiML).

The result is Twilio automatically connecting me with the phone number listed on the Craigslist post. This is a derivation of a very popular Twilio use ‘Click-to-call’ that many lead management and sales teams use to better engage with their customers.


Now I don’t have to be on Craigslist 24/7 to find a suitable apartment listing; I can just wait for my web server to send out emails or phone calls to schedule appointments with Craigslist posts that fit our preferences. Now if there were only a way to automate the showing up, preparing the paper work, competing against other bidders, and the remainder of the apartment leasing process.


Tagged , , , , , , , , , ,

Phantachat: node.js, websockets, and ephemeral conversation

In person conversation is fleeting. Words hang in the air for a brief moment then vanish suddenly. All participating members have to be present and aware in order to keep up with the conversation.

Conversation online has traditionally been very asynchronous. Instant message and email both log your messages with a timestamp, allowing you to maintain a history conversations for you to browse afterwards. You can message your friend knowing that she probably won’t respond immediately.

Phantachat is an online chat web app, but tries to mimic chat in person. Your messages linger for no longer than a few seconds, forcing you to have that page open on your screen to be able to follow the conversation. Messages are also sent in immediate real time, without having you to hit enter, and there is no backspace.


Websocket is a web technology providing simultaneous two-way communications channels over a single TCP connection. This allows for immediate, seamless, synchronous data passing between the client and the server. We decided that in order to get the chat to behave as real time as possible, that we would choose to use Websockets (we went with Socket.io with Node.js/Express.js).

Unfortunately, there are very few free web/application hosting services out there that support websockets. Heroku doesn’t support web sockets, but you can configure your application to instead use long polling, which allows the client to send HTTP requests to the server at regular intervals and immediately receives a response.

Some other hosting platforms out there (www.appfog.com, http://www.dotcloud.com, http://www.nodejitsu.com) may support web sockets, but unfortunately they don’t provide a free tier. Our next plans are to spin up an instance on AWS so we can deploy a solution that uses websockets instead of long polling.

Dynamic chat rooms

In order to generate a unique chatroom, we created added some logic to the routing: if someone goes to the root, then generate a new hash (I used the hashids package for generating codes) and redirect to the new route with the hash at the end and start a new socket.io namespace; if someone goes to a hash that exists, then throw that person into the same socket.io namespace; and if someone goes to a route that is a hash that doesn’t exist, the server will return an error page.

With socket.io, there are two main ways of creating ‘chatrooms’: using ‘rooms’ or using ‘namespacing’. We decided to use namespacing, because namespaces can be connected to by the client (but only if it already exists on the server). With rooms, they can only be joined on the server side.

When a new namespace is created (when a new hash is generated on the server), the server creates a new namespace, then passes the hash to the client to allow the client to join that specific namespace.

Next Steps

This is still a very simple app. Next steps could be sharing other types of media that will also vanish within a few seconds (images, for instance). Or to allow more than two people to join each chat room (right now, it is only one on one).

Suggestions, feedback, and new ideas are always welcome!

Github: https://github.com/lambtron/phantachat


Tagged , , , , , , , , , ,

Welcome to the A-list

Note (5/31/2013): Recently, this app was featured in Lifehacker. Kindly note that this app is still in beta and there are probably many bugs that have yet to surface themselves (one such obvious one is checking in multiple passengers, as I wrote this for me and my buddies and didn’t have children to buy tickets and for whom to check-in). If you do run into any issues or have any feedback, feel free to tweet at me @andyjiang or comment on this blog post and I will make a note of it.


Recently, a conversation over dinner about personal life hacks without the direct intention of blatant commercialism spawned a neat idea that scratches an itch that us west coast jet setters face occasionally—not only just checking into Southwest flights, but also checking into those flights as early as possible to attain a highly coveted ‘A-list’ boarding group.

There have been too many times when the 24 hour period prior to the flight occurs when I’m away from my computer. This web application, however, will check into your Southwest flight for you.

Instead of going to a website where you input the information necessary for you to check in (name and confirmation number), you now can you just forward your Southwest flight confirmation email (the one with the subject that says ‘Your trip is right around the corner!’) to sw@to.andyjiang.com. Then, when the 24 hour check in window opens up, the web application will check in to the flight for you.

How does the app work?

The Mandrill (hosted by Mailchimp) server receives an inbound email and sends a POST request to the web server. The web server then extracts the text from the body of the email from the POST request and runs several regular expressions on it to get the following pieces of information: first name, last name, confirmation number, your email address, the URL it needs to go to in order to check in for you. It saves this data into a Mongo database (MongoHQ add-on with Heroku).

An hourly task runs on the server (Heroku Scheduler) to look at all of the records in the mongo database; for each record, if it is time to check in, the web server  uses the Mechanize library to get and parse the HTML of the confirmation web site, fill in the required information, and check in for you. Then it takes your boarding group and boarding position and sends you a confirmation email.

Tools I used

Since debugging is 90% of any project, it is important to learn to break the project into small, achievable goals, to quickly isolate problems and to iterate through working solutions. Below is a list of some of the tools I used to test quickly and build out the application.

RubularRegular expressions were critical to parsing the body of the inbound email. This nifty website allows you to test all of your regexp patterns against Strings of your choice. Combined with IRB and .class?, you can very quickly determine the right commands to extract the information needed from the text.

LocaltunnelLocaltunnel, hosted by Twilio, allows you to expose an endpoint on your local server to receive HTTP requests from other servers. This is important when you are using a service that will send you webhooks (Twilio, Mandrill inbound emails, etc.), so you don’t have to push your changes to a production server in order to test whether your app correctly processes the POST request to your endpoint. With Localtunnel, you just follow the instructions on the site, copy and paste the localtunnel URL/endpoint generated (tied to your local server), and provide it to the web service so it knows the address to send its request, thereby allowing you to test locally.

View Elements in Developer Tools: In Chrome (Firefox and Safari, also), you can view the HTML document of each page you are on by going to View > Developer > Developer Tools (or Option + Command + I). This is important when you are using Mechanize, which allows you to parse HTML given the HTML element, id, class, name, etc. I would test code by using IRB, initialize a Mechanize object, open up the browser, and navigate the web from the terminal.


Feedback is always welcome. If you think you are going to forget to use this next time you fly Southwest, go into your Gmail account right now and set up an instant email forwarding. Go to your Gmail’s Settings > Forwarding and POP/IMAP > Forwarding, then select Add a Forwarding Address. Type in “sw@to.andyjiang.com” and send the verification email that will contain the confirmation code.

Forwarding and POP/IMAP > Forwarding to set up a new email address to auto forward your Southwest itinerary emails" class /> Go to your Gmail's Settings > Forwarding and POP/IMAP > Forwarding to set up a new email address to auto forward your Southwest itinerary emails.

Forwarding and POP/IMAP > Forwarding to set up a new email address to auto forward your Southwest itinerary emails” class /> Go to your Gmail’s Settings > Forwarding and POP/IMAP > Forwarding to set up a new email address to auto forward your Southwest itinerary emails.

You will receive shortly an email that will provide you with the confirmation code that you can enter in the above field. The return email will also include a link that you can just click on to verify and confirm sw@to.andyjiang.com as the new forwarded email address.

The generated email with your confirmation code to verify the auto forwarding email address

The generated email with your confirmation code to verify the auto forwarding email address

Once you confirmed the email address to allow you to auto forward to it, then you will have to set up the filter for the right emails to be auto forwarded. Click on the link in the tip ‘You can also forward only some of your mail by creating a filter!’ and then either copy and paste “from:southwest subject:(Your trip is around the corner)” into your Gmail search bar or use the template in your ‘Show Search Options’ drop down below.

Setting up autoforward for any Southwest itinerary emails to sw@to.andyjiang.com

Setting up autoforward for any Southwest itinerary emails to sw@to.andyjiang.com

Creating the filter to auto forward specific emails to sw@to.andyjiang.com

Creating the filter to auto forward specific emails to sw@to.andyjiang.com













That should set it up so that any future email from Southwest with that particular subject will be automatically forwarded to my web app, which should hopefully check you in so you won’t have to sit in the back between the two fat, sweaty guys.

I will blog more about the actual code in future posts.

If you have any questions, please let me know!


[EDIT]: Please note that this is still not 100% battle tested, but it will improve over time the more emails it receives! Thanks, everyone!

Tagged , , , , ,

5 signs you are a silicon valley startup engineer

it has only been a few weeks since i have moved into the fast paced world of startups and engineering from that of the financial and capital markets.  however, there are just some hard truths that i had to wrap my mind around in order to become fully assimilated in this new universe.  here are five inarguable signs that occur when you are in the process of becoming into a silicon valley startup engineer.

1. less time spent on LinkedIn, more time spent on Github: LinkedIn is the professional social network of choice, because it is conveniently sandboxed from the rest of your virtual existence.  as such, the pictures from last night’s cat-burglar-themed orgy do not get automatically updated to  your profile (that and the fact that LinkedIn’s interface is way confusing with its ‘degrees of separation’ among other stuff that are the only differences from popular social networking tool Facebook).  However, LinkedIn profiles are prone to the same problem that normal resumes have: unbridled embellishment.  Your two week stint at your dad’s brother’s dental practice where you LOL’d at geekologie.com all day is incidentally a paid internship program where you helped manage multiple client accounts and successfully improved operational efficiencies (measured by several multiple-syllable data metrics) through method automation and strong analytical skills.  Engineers, however, are builders by profession and self-proclaimed statements infused with marketing buzzwords such as “created synergies through boosting aggregation operation logistics” have no positive effect.  Github is the place where developers keep their portfolios of cool things they have made; it is examples of their past work that demonstrate their abilities.  Engineers normally have githubs on their personal splash pages and business cards that are exchanged in person.  Some people even exchange github links prior to making eye contact (i have seen it done).  An engineer without a github is a young urban professional without a self-entitled attitude.

2. standards for women lowers drastically:  silicon valley is a black hole of attractive women. it could be the dense population of computers that largely inhabit a typical man’s time that could be the major deterrant for women to flock to silicon valley (computers do as their told and only requires a push of a button to turn on, especially if using a mac).  you’ll realize you have reached silicon valley when you start to notice when there is a female within the nearest 10 mile radius.  you’ll examine it quizzically, questioning its existence (‘is that really a chick?’), then followed by admiration (‘wow, i really respect her tenacity to be doing what shes doing’), then followed again by a mix of curiousity/fascination (‘what is her story? she probably took a wrong turn somewhere.’), then finally followed by skepticism (‘she is most definitely fake; i’ve seen real girls on the internet and that is definitely not it’).  moreover, your standards teeter on the edge of plummeting down a endless abyss.  you’ll start overlooking underbites and lazy eyes.  a symmetrical face is a godsend.  although there are few women here, it is definitely less distracting–had there been more physical women in silicon valley, the growth and innovation would grind to a screeching halt.

3. develop a strong affinity for white boards:  if silicon valleyers had it their way, all of the asphalt on the road would be replaced with shiny white boards.  because who doesn’t like a surface on which items can be most conveniently read/written/re-written?  startup offices are most often judged on the following two criteria: the abundancy of snacks and drinks, and the size and girth of white board space.

4. wardrobe shifts to cargo shorts/pants and running shoes:  not quite sure if this is the cause or the result of lack of women in the valley, but wardrobes tend to shift towards the standard engineer gear: hoodie, cargos, and running shoes. its the practical approach to attire: it is comfortable and those around you (people, computers, machines) don’t care what you wear.


Tagged , ,

Bay Area vs. New York Metropolitan Area


Note: Pac Heights and Nob Hill is also similar to West Village (suggestion by Alvin Yeh).

The overwhelming conclusion of San Francisco (“SF” or “the City”) coming from New York City, at the risk of generalizing, is that everyone is a hippie (the original 1.0 of today’s hipster*, the hipster that actually gave a shit about society)–nature loving, recycling, vegetable-eating, pabst-drinking, american spirits-smoking, and large glasses/cardigan/skinny jeans-wearing.  there are plenty of hipsters, as well. 

i didn’t directly compare manhattan and SF because there are so many neighborhoods in manhattan that i couldn’t find in SF (please correct me if i’m wrong, as i haven’t been in the bay area long enough to form any lasting opinions). also, this was meant to be more of a jab at SF for just having a lot of hippies and hipsters.

While the general SF populace consists of these progressive lifers, there are various pockets of communities that vary around the average SF hippie: by household income (usually determined by the amount of organic ingredients on the local restaurant’s menu), by age, and by alcoholic preference (wine or beer enthusiasts).  regardless, everyone respects everyone elses right to happiness, which is a beautiful thing.




*today’s hipsters, over the past few years, have achieved a cultural identity within the eyes of america.  they are loosely tied together from their aesthetic appearance: thrift clothes, rolled-out-of-bed haircut, and a general vibe of grunginess. while other counter culture groups identified by fashion distinctions, it appears that fashion is their only distinctions, besides general apathy towards life outside their immediate surroundings and enjoying various bands and artists that are so underground they haven’t written their first songs yet.

Tagged , ,