www.archive-net-2013.com » NET » C » CODERANGER

Choose link from "Titles, links and description words view":

Or switch to "Titles and links view".

    Archived pages: 8 . Archive date: 2013-07.

  • Title: Noah Kantrowitz
    Descriptive info: .. Noah Kantrowitz.. Writing.. Code.. Thoughts.. Contact.. June 2, 2013.. Arrays and Chef Attributes.. February 20, 2011.. LazyProfile middleware.. January 16, 2011.. Using SELECT FOR UPDATE in Django.. December 1, 2010.. Writing DSLs in Javascript.. November 19, 2010.. Chef slides.. November 14, 2010.. System monitoring: A play in 5 acts..

    Original link path: /
    Open archive

  • Title: Contact
    Descriptive info: Email.. Twitter.. @kantrn.. Facebook.. coderanger.. IRC.. coderanger@freenode.. XMPP.. LinkedIn.. Google+..

    Original link path: /contact.html
    Open archive

  • Title: Arrays and Chef Attributes
    Descriptive info: June 2, 2013.. A conversation with a friend today reminded me how easy it is to find unexpected behavior with Chef attributes when using array values.. Arrays are good for many things, and are often a seemingly natural fit to describe server values.. The example my friend was having issues with was:.. default.. [.. chruby.. ][.. rubies.. ].. =.. jruby.. ,.. 1.. 9.. 3.. 2.. 0.. 0.. node.. each.. do.. |.. ruby.. end.. This is simple, consise, and relatively unambiguous to the reader; all hallmarks of good code.. There is a sinister side though, how do you cope with merges on an array? Chef attributes exist in a.. nested structure.. , where different sources and precedence levels are.. merged together.. to form the final attributes that your recipes run against.. With hashes, this merge is relatively straight forward, if both the original and override values are hashes they are recursively merged together all the way down.. This logic is less clear on arrays though.. In Chef 11, the behavior is that on different precedence levels, the higher value overrides as with strings and other simple types:.. override_attributes.. ({.. =.. {.. 8.. 7.. }.. }).. ==.. However things are not as simple when the two values are at the same level.. The two arrays  ...   overriding are clearer.. You can also either add or remove a value at any point in the tree:.. false.. This both clarifies your existing code, and allows flexibility you may need in the future.. Sometimes you really do just want an array in the end, perhaps to pass to an external library, or to render into a template:.. inject.. (.. [].. ).. memo.. key.. value.. ;.. But what about order?.. So one crucial difference between using an array and a hash of boolean flags is a loss of ordering.. Ruby does track the insertion order in hashes, so usually the final order of keys will follow the default/normal/override ordering that attributes themselves use, but sometimes this is not enough.. In these cases we can instead use a hash of weight values, which we sort on afterwards:.. 100.. 50.. \.. sort_by.. This keeps all the earlier benefits of being able to change or remove values at any point in the precedence tree, while still getting consistent ordering.. All in all these patterns do involve both a little bit more Ruby code, and some more careful planning, however the immediate benefits make it worth a look and your maintenance engineers will thank you in six months.. Please enable JavaScript to view the.. comments powered by Disqus..

    Original link path: /2013/06/arrays-and-chef/
    Open archive
  •  

  • Title: LazyProfile middleware
    Descriptive info: February 20, 2011.. The topic of extending the User model in Django came up recently on.. Convore.. In the discussion I mentioned a minor hack we had come up with at work to reduce the friction of working with user profile models (which are still the safest way to add extra fields to User)..  ...   adds a.. user.. The only point of warning is that also like AuthenticationMiddleware, this cannot be enabled for only one site in a multi-tenant deployment scenario.. class.. LazyProfileMiddleware.. object.. ):.. Middleware to attach a lazy.. profile value to all requests.. lazy_profile.. property.. lambda.. self.. :.. user.. get_profile.. ().. else.. None.. def.. process_request.. __class__.. profile..

    Original link path: /2011/02/lazy-profile-middleware/
    Open archive

  • Title: Using SELECT FOR UPDATE in Django
    Descriptive info: January 16, 2011.. A frequently overlooked feature of SQL is the.. FOR UPDATE.. clause on.. SELECT.. Even in this age of NoSQL datastores, most large-scale webapps will use a relational database somewhere, and good money would bet on most having a transaction race condition somewhere.. Transaction isolation in a nutshell.. Transaction isolation is the mechanism by which a SQL database prevents multiple in-progress transactions from interfering with each other.. Postgres provides two different levels,.. read committed.. and.. serializable.. Read committed.. is the default and is sufficient for the vast majority of operations.. A simple mental model for how this works is to have a dirty flag on each row.. Any time a row is touched by an.. UPDATE.. or.. DELETE.. statement, the dirty is set.. Similarly any time a.. is about to operate on a dirty row, more specifically one which was marked dirty by another transaction, it will block and wait for the other transaction to either commit or rollback and will then update the value it was trying to access.. Using UPDATE.. As long as you can express all your operations using SQL expressions in an.. , this isolation is sufficient.. For example, you can increment an integer column via.. UPDATE foo SET x=x+1 WHERE id=1;.. Unfortunately Django doesn t make this easy.. The naive way to perform the same operation is:.. obj.. Foo.. objects.. get.. id.. x.. +=.. save.. This means that two requests can come in simultaneously and the second will clobber the first:.. A.. # obj.. x is now 1.. B.. x is now 2.. x saved as 2, as expected.. x  ...   procedures, but this is effectively impossible to do while keeping mutli-database compatibility.. The other option is to do the computation in Python code.. Without.. this puts us back into race-condition territory in the same way as.. Model.. save().. What.. does is to set the same dirty flags that.. use before returning the rows.. This means that no other transaction can alter it.. Unfortunately the Django ORM doesn t.. yet.. expose.. as part of the query system, but with some creativity we can add it in.. Much of the credit for this goes to.. Alexander Artemenko.. who wrote the initial version of the helper.. from.. django.. db.. import.. models.. connections.. models.. query.. ForUpdateQuerySet.. for_update.. sqlite.. in.. db.. settings_dict.. ENGINE.. lower.. ():.. # Noop on SQLite since it doesn t support FOR UPDATE.. return.. sql.. params.. query.. get_compiler.. as_sql.. model.. _default_manager.. raw.. rstrip.. FOR UPDATE.. ForUpdateManager.. Manager.. get_query_set.. using.. _db.. Then all you have to do is inherit from.. in your manager and use.. for_update().. at the end of the filter chain:.. qs.. 0.. something_complex.. ()).. An example.. Our main use case at Atari for all of this is updating a user s billing information, specifically their next billing date for people on monthly cycles.. Due to the nature of MMOs, it is possible that someone could redeem a game-time card at the exact moment that a background task is processing them for monthly billing.. At heart this code boils down to:.. User.. ()[.. new_date.. next_billing_date.. relativedelta.. months.. day.. next_billing_day.. This is both transactionally safe and concise, which is just about all you can ask for from a SQL database..

    Original link path: /2011/01/select-for-update/
    Open archive

  • Title: Writing DSLs in Javascript
    Descriptive info: December 1, 2010.. Beginnings.. Recently at work I have been building a testing console for our web APIs.. It began very simply, largely influenced by the excellent.. Twitter API console.. This design rapidly descended into a sea of hacks.. The.. Hurl.. style works well for RESTful, or nearly RESTful as in Twitter s case, APIs however it was not as effective for our primarily JSON-RPC based system.. My next approach was to make something more terminal-like, based on the.. WTerm.. jQuery plugin.. This mapped better to a procedural API, but it still felt very limited.. Specifically it needed variables.. eval().. ^w.. Given that this was a web site I already had the not insubstantial power of Javascript at my disposal.. My first thought was that I could dynamically generate functions to map to our API, and then simply use eval() to process input.. This had two major problems; one, I couldn t isolate the namespace of the terminal from the rest of the page and two, I had no way to deal with the asynchronous nature of the AJAX calls.. To fix the first problem I tried several variations of restricting the execution environment of eval(), but I was unable to find an acceptable, cross-browser solution.. To deal with the second I tried various attempts at code generation from the input strings, but again I left empty handed.. All in all, while Javascript is a very powerful language for scripting pages, it isn t the right tool for embedded DSLs.. Parser 1.. Once I had decided down the path of writing a custom DSL, my first thought turned to parsers.. WTerm is already a DSL of  ...   Operator Precedence.. parsers.. The example code is conveniently in Javascript, so I set out to adapt it into a working language.. The provided parser is pretty much a working implementation of the bits of Javascript that I needed out of the box, so only a few modifications were needed.. For starters I changed how name tokens are handled in.. advance().. o.. scope.. find.. v.. define.. t.. This means that any name token will be created in the scope it appears in.. I also didn t want things like flow control or functions so I altered the returned parser to use:.. expression.. This provided a solid base to build the DSL on, with Javascript literals ( numbers, strings, objects, and arrays), basic math operators, attribute access, and simple variables.. Interpreter.. Crockford s code only provides a parser, the other half is to have something to execute that parse tree.. Below is a snippet of my callback-based function:.. switch.. arity.. case.. literal.. cont.. name.. variables.. err.. Variable \.. \ not defined.. ]);.. binary.. interpret.. second.. function.. first.. },.. +.. value1.. value2.. It isn t the fastest thing on the block, but it is perfectly adequate for interactive usage.. Putting it all together.. There are still some bits of complexity to hook up the RPC calls to the interpreter, and in my case setting up OAuth from the browser as well, but once you are this far down the rabbit hole they will be a walk in the park.. If you are thinking about building a DSL into a web page, I hope this has pointed you in the right direction, or at least a slightly less wrong one..

    Original link path: /2010/12/writing-dsls/
    Open archive

  • Title: Chef slides
    Descriptive info: November 19, 2010.. I will put together a longer post on.. Chef.. later, but below are my.. slides.. from the SF Django meetup..

    Original link path: /2010/11/chef-slides/
    Open archive

  • Title: System monitoring: A play in 5 acts
    Descriptive info: November 14, 2010.. Act 1: Munin.. Munin is dead simple to configure and deploy, and has a very simple, modular sensor API.. It uses a collection agent so it can easily monitor values not readable over the network (which is a hard requirement for me).. Beyond that it is really just a small wrapper over RRDTool.. Its graph UI is painfully simple and it has only the simplest of alerting systems.. Act 2: Munin and Nagios.. Nagios is the gold standard when it comes to FOSS network monitoring.. It is primarily aimed at alerting on failures, so Munin is still needed for logging time sequence data like performance metrics.. This provides more reliable reporting, but it comes with several downsides.. One option is to simply duplicate any required sensors between Munin and Nagios.. Another is to use the Munin data to trigger alerts in Nagios.. This prevents the duplication, but you are still stuck with Munin s UI for graphing.. Act 3: Nagiosgraph.. There are several addons for Nagios that add time-sequence logging,  ...   as a potential replacement for Nagios, with much simpler configuration and UI.. The downside is it doesn t seem to have any support for logging performance data, so it is a non-starter for now.. Act 5: Opsview.. Opsview is built on top of Nagios and Nagiosgraph.. It provides some minor UI enhancements as well as some major performance fixes (which I am unlikely to ever notice with ~10 machines to monitor).. It also provides somewhat simplified configuration, though the full power of Nagios is available if you dig in to the system.. Conclusion.. Opsview seems to be no worse than vanilla Nagios, and it does provide a slightly nicer base to build from.. Overall this does feel like picking the lesser of several evils though.. The only well-known entry in the Python world is Zenoss, which has minimal remote monitoring capabilities and seems to be even more complex and convoluted than Nagios.. Munin 2.. 0 does show some promise as a future option, but for now Opsview looks like the way to go..

    Original link path: /2010/11/system-monitoring/
    Open archive



  •  


    Archived pages: 8