uKeeper - from link to full page

Version 0.16.2

2013-03-29T17:52:00-05:00

A lot of work on the back-end around RMQ/MongoDB/MultiNAS related support. Such ability helps to build and maintain rapidly and flexibly distributed core of the system.
Added async distributed event bus. This addition improved latency of metadata updates and made everything much more consistent. For example, no more “eventually consistent user’s email update” which caused sometime a few seconds delay until new/updated user’s info distributed across all uKeeper nodes.
Introduced new modular composition allowing safer and faster release of new functionality.
Added support for Reddit pages
Added support of Surfingbird capturing
Fixed processing of “base href” which resolved number of issues with missing images.
Other, minor bugfixes.

version 0.15.3

2013-01-31T16:45:00-06:00

Improved processing of twitter’s statuses
Fixed processing of twitter’s pics
Improved image extraction for referrer-sensitive sites
Added support of %i and %p for XMPP (jabber) links
Workaround for the incorrect title returned by external parsers
Improved fault-tolerance in non-S3 (alternative) storage
Added experimental RMQ–based backend

version 0.15.2

2013-01-09T17:59:00-06:00

This is mostly bug-fixes and internal improvements release.

Added ability to change S3 regions
Implemented pessimistic strategy for data retrieval from S3- Added backup shared storage method, eliminating S3 completely in case of need
Improved support of official mobile Twitter client for both “Mail Tweet” and “Mail Link”
Fixed issue with images referenced by partially invalid urls
AWS SDK updated to the current version
If sites doesn’t allow image to be downloaded uKeeper will keep the original link
Some minor performance improvements

AWS issues

2013-01-04T14:17:00-06:00

Since yesterday I see exceptionally high ratio of failed or significantly delayed S3 operations. As you may know, uKeeper internals runs on AWS and make use of several services like S3, SQS, DynamoDB and some others. S3 especially important for distributed data retrieval process. Practically, all article-related resources like images and other attachments coming from S3.

S3 works as eventually-consistent storage, which means a simple thing - in some cases the data wrote to S3 not immediately visible to all readers, which is fine if readers expects such behavior and could wait enough time till updated data available. uKeeper one of such smart readers and worked this way since day one. Usually, the waiting period was 100-200ms, in very rare cases - 500ms. As a paranoid developer I put in place a waiting period for up to 15sec, but since Apr 2012 I have seen 4 cases only with unusually high latency (~3sec).

However, since yesterday S3 has been acting differently - relatively large part of submitted objects still not available for hours after the write! I have informed AWS support, and looks like they are trying to fix it. I see number of such incidents decreasing dramatically and for last 12 hours I got just 4 delayed writes.

On uKeeper side this issue initially caused “request rejected” to some users. As soon as problem was detected I put in place a hot-fix allows to process articles even if one of resources failed / delayed by S3. In this case user will get an article, but it may have a missing picture. Please note - this is really, really rare case now and hopefully AWS will get it fixed completely very soon.

From this indecent I learned a few important things about “what to do if AWS acting strange” and going to implement a new set of backup strategies for cases like this.

UPD: 01/05 13:57 CDT – The problem with S3 was resolved completely.

version 0.15.1

2012-12-31T14:12:00-06:00

Added ability to catch pages in PDF format. Adding %p to subject will do the trick.
Added ability to catch pages in image format (png) – %i to subject
Added new, regexp-based rules for deeper url match. It allows to tweak parsing for the different parts of the same site if needed.
Added detection and extraction of embedded youtube video
Added support for non-latin URLs
Implemented special parsing for twitter’s conversations
Added support for twipic images and surrounding text
Fixed incorrect detection of image type in Evernote
Fixed issue with relative image URL’s without leading /
Fixed issue with urls with “+” in web catcher
Improved instapaper & pocket forwarding
Header shows expanded url, not the short one

both PDF and PNG generators based on wkhtmltopdf

version 0.14.6

2012-10-31T02:23:00-05:00

Added link forwarding for instapaper and pocket destinations
Fixed extraction of multiple tags
Improved processing of binary attachments
Preview for cases with multiple URL matches
More space-related URL’s fixes
Some other improvements, mostly performance related

version 0.14.5

2012-10-16T13:24:00-05:00

Workaround for Evernote’s issue caused pdf dups.
Added special parser and link converter for google-groups.
Added ability to catch pages/links preprocessed by instapaper or readabilty.
Better support for title meta-tag. In some cases should address duplicated titles and generally makes title extraction smarter and more predictable.
Fixed incorrect tag’s detection from subj with links (empty body case).
Improved XMPP link detection for adium client.
Fixed parsing of urls with “+”
Fixed processing of links with relative path like “../../”
Proper cleaning of unicode’s non-breakable spaces in urls

version 0.14.4

2012-09-24T18:21:00-05:00

Added new jquery selector-syntax for the new custom parser
Added support of custom titles for selectors-based parser
Special support for new, dynamic blogger’s themes
Improved resiliency for most of S3-based operations
End-to-end fully automatic monitoring systems for regular and full capturing
Fixed processing of twits with image only and with internal redirects’
Fixed reporting and base processing for links with hash character
Migrated to latest stable Netty 3.5.5
More fixes for problematic redirects
Simplified download tasks manager
Custom parsing settings available for more testers

version 0.14.3

2012-09-02T14:41:00-05:00

Added support of multiple links via jabber (up to 10 links in a single message)
Added ability to disable extra redirects for some sites, forced login in such cases
Proper support for existing custom rules in preview-rules/edit-rules forms
Implemented better concurrency for page preview (custom parser)
Optimized downloading of page resources (images and others embedded elements)
Better support of reply-to filed
Fixed an issue caused usage of email alias instead of correct email in some rare cases
Improved extraction of pages with multiple redirects
Fixed processing of parametrized and dynamic images

version 0.14.2

2012-08-14T18:42:00-05:00

Internal improvements, mostly around custom parsing.

Added ability to change User-Agent for custom sites. For instance it allows to switch (in case of need) to mobile version.
Added generic url-converter for custom url mapping. This is useful for tricky sites where the article link should be derived.
Excluded element in custom parsers can be attribute value in addition to class name and id.
DKIM was disabled for now, seems to make some sort of incomplete signature
Development migrated to IntelliJ IDEA

version 0.14.1

2012-08-07T17:32:00-05:00

Added optional multi-link support. To turn it on put ’^’ at the begining of your subject.
Added DKIM (DomainKeys Identified Mail) singature to all ukeeper’s emails. Should make our emails treated better by anti-spam filters.
Added support of basic authorization for incoming links. It still a very bad idea to email your user:password, but if you really have to - ukeeper will process it.
For the most forgetful users - if you didn’t put any link, either in subject or body, ukeeper will try to do a basic search for you, based on provided subject.
Added support of user-defined parsing rules. Not actived yet for most of users, just for a few selected beta-testers.
Imporved detection and processing/expanding of short urls.
Added unlimited depth for parents in custom parser’s mathcing.
Added support of aliases for the most popular email services.
Some minor outlook-related fixes.

version 0.12.6

2012-07-20T21:34:00-05:00

improved processing of short links. For some cases such redirect caused encoding lost and prevented custom parsing.
added special extraction for dzone.com links
improved detection of slow pages and implemented timeout auto-adjustment
fixed a rare problem with incorrect title for persistent subject mode
preparation for user-defined parsing parameters with live preview

Support changes

2012-07-13T19:52:00-05:00

We have migrated to uservoice for all type of ticketing, support and sharing new ideas. Feel free to hit us with your bright ideas and sad bug reports.
Report error link will open a simple form where you can add description and your name (optional). This form integrated with our new ticketing system.
Online chat proved to be just a useless toy - removed.

version 0.12.5

2012-07-12T21:47:00-05:00

Added ability to use custom parsers in FULL mode
Improved compatibility with some websites rejected ukeeper’s image requests
Correct processing for image/resource’s url with mixed slash and backslash
Custom parser made smarter and can detect first match
Added support of emails with “+”
More fixes for empty titles
Fixed web-ukeeper for urls with “&”

version 0.12.4

2012-07-06T20:13:00-05:00

improved accuracy of Outlook’s links processing. In some cases silly Outlook decided to split your link.
support of exotic charsets simplified and unified.
resolved problems with content and subject in different encodings in forced ! mode.
in case if page doesn’t have any title and title can’t be extracted from the content uKeeper adds site’s domain as a title.
resolved problem with non-latin urls, with special characters.
increased timeout for data extraction to allow slow sites and/or big requests to be processed.
fixed processing issue for some links with double-quotes.
added ability to customize article extraction in the new parser on the fly.

The last item is extremely interesting one. This is continuation of “very experimental article extractor” introduced in v0.11 and this extractor allows to adjust uKeeper’s behavior on micro-level. For example, making good-looking–in–your–mailbox article from some exotic site, with very unusual layout, can be done in a minute. It also allows to handle known “problematic” sites with minimal article/content but with a lot of valuable comments. And one more thing - this custom extractor could be even user-defined some day, and you will able to specify special behavior (processing) for your favorite site(s).

Version 0.12.3

2012-06-29T17:39:00-05:00

Fixed a bug prevented processing of short, case-sensitive links, introduced by 0.12.2
Added support for some upper-case links supposed to be lower-case
Added smarter processing for Twitter statuses. uKeeper will try to extract link from the status, if presented.
Added ability to implement custom preprocessors for predefined sites. It will allow all sort of interesting tricks, for example I will able to request for maps sites print-friendly version instead of the regular so-hard-to-get-data-from.

Version 0.12.2

2012-06-28T14:43:00-05:00

Added a new way to send links - via jabber. See Extras “uKeeper via Jabber”. Not fully polished yet, but works.
Fixed problem with “(” and “)” in links prevented correct processing.
Added ability to capture youtube links.

version 0.12.1

2012-06-20T18:59:00-05:00

This is sort of significant update with massive improvements in data extraction. Version 0.12.1 added several different methods of article detection and it combines all of these method to get clean and complete page.

Improved accuracy of article’s detection and extraction.
Added ability to process multi-page articles.
Added support for the new binary formats - doc, xls, ppt, vsd, vst, zip, gz, tgz.
Added ability to adjust extraction rules manually. It should allow almost instant fix for reported problems.
Added multi-step analysis of extracted data with automatic tunning, if necessary.
Smarter detection of article’s title.
Added initial content extraction from emailed tweets. So far tested just on a few iOS clients. Your feedback will help to make it right.
Fixed an issue with broken formating for some extracted pages.
Fixed an issue with partial articles for some links.
Improved processing of non-article/index pages, like search results and so on.
Fixed procession of links with spaces.
Fixed incorrect support link in ukeeper’s error email.
Fixed lack of error email in case if response too big and can not be send.
Disabled support for multi-line links until I have some smart way to handle it without affecting pure, single-line links with some content below.

version-0.11.1

2012-06-15T16:18:00-05:00

Added initial version of very experimental article extractor. For now you can try it by adding “%” at the start of the subj. It’s buggy and fresh, but has a great potential.
Resolved issue with double-quotes in the encoding info sent by some crazy servers. In rare cases it caused “reject email” from uKeeper
Resolved problem with multi-line links. Seems like it was mostly outlook’s thing.
Optimized downloading and processing of embedded pictures. The same image won’t be processed multiple times and won’t be attached multiple times anymore.
In case, if header’s encoding different from meta info, ukeeper will try to autodetect it based on html, and if this detection match header or meta - this one will be used.
Added automatic code deployment to all worker’s nodes
Created a template for the new worker’s node and automation for quick initial build.

version 0.10.2

2012-06-10T19:09:00-05:00

responses migrated to SQS + S3. This allows more reliable delivery and higher redundancy. Also it makes simpler to scale ukeeper out.
added support for partially defined charset and some wrong, but widely used aliases.
improved binary email for images.
added auto-tuning for workers and several protections against excessive load.
added protection against multiple submission of the same request by the same user in short period of time.
fixed an issue with multiple error messages sent out in some rare cases.
message polling made less aggressive and more efficient.
other minor fixes