extracting images from hangouts

December 29, 2016
bullet
grep, hangouts, jq, sed, wget

Download images from a google takeout dump of hangouts. Requires: jq, grep, and wget.

If you want all media, you can just use grep for image_url, clean up the json remnants, then download with wget -i

If you only want a single conversation and the other user's images, you'll need to use something like jq to extract the relevant hangout events before grepping.

cat Takeout/Hangouts/Hangouts.json | \ jq -c '.conversation_state[] | \ select(.conversation_id.id=="CONVERSATION_ID") | \ .conversation_state.event[] | \ select(.sender_id.gaia_id=="SENDER_ID")' | \ jq . | grep image_url | \ sed 's|^\s*"image_url": "||g' | \ sed 's|",$||g'

Setting up a raspberry pi with squeezelite

September 18, 2015
bullet
dlna, mpd, raspberrypi, squeezelite, subsonic
  1. Acquire raspberry pi
  2. Install raspbian on SD card per https://www.raspberrypi.org/documentation/installation/installing-images/
  3. Update the firmware rpi-update
  4. FIXME Remove unnecessary packages: apt-get remove --purge scratch dillo xpdf galculator lxde-common lxde-icon-theme hicolor-icon-theme wolfram-engine
  5. Adjust swap size in /etc/dphys-swapfile: CONF_SWAPSIZE=100 otherwise your card will fill up.
  6. Update the OS: s/weezy/jessie/g in sources.list, then apt-get update && apt-get upgrade && apt-get dist-upgrade (wait forever)
  7. Set the default sound output to headphone jack: amixer cset numid=3 1 per https://www.raspberrypi.org/documentation/configuration/audio-config.md (if testing with omxplayer, note that it ignores this setting, so you'll have to use -o local)
  8. Install squeezelite apt-get install squeezelite
  9. Add buffering parameter in /etc/default/squeezelite: SB_EXTRA_ARGS="-a 160" per https://code.google.com/p/squeezelite/issues/detail?id=28
  10. Eh, fuck LMS! Let's try subsonic...
  11. Install subsonic
  12. Enable DLNA (premium!!?!?!?!) in settings
  13. Add library folders
  14. Enable access to library folders for guest account
  15. Install GinkgoDlna on phone
  16. Install some things for gmediarender: sudo apt-get install libupnp-dev libgstreamer0.10-dev \ gstreamer0.10-plugins-base gstreamer0.10-plugins-good \ gstreamer0.10-plugins-bad gstreamer0.10-plugins-ugly \ gstreamer0.10-ffmpeg \ gstreamer0.10-pulseaudio gstreamer0.10-alsa
  17. Install gmediarender sudo apt-get install gmediarender
  18. Use GinkgoDlna to queue audio for playing
  19. Eh.... fuck it, let's try MPD per http://www.hackerposse.com/~rozzin/journal//whole-home-pulseaudio.html or https://fruit.je/mpd-rtp
  20. On the NAS/server: apt-get install pulseaudio rtkit mpd
  21. Edit /etc/mpd.conf and /etc/pulse/default.pa per the site
  22. Add a controller: sudo apt-get install ncmpc
  23. sudo chown mpd.audio /var/lib/mpd/music
  24. delete the fucking tag db to make it reload the mp3s rm /var/lib/mpd/tag_cache; supposedly it should work with mpc update
  25. Fix the fucking routes: sudo route add -net 224.0.0.0 netmask 240.0.0.0 dev eth0
  26. On the pi: sudo apt-get install pulseaudio rtkit
  27. Edit /etc/pulse/default.pa per the site
  28. Make /etc/network/if-up.d/pulseaudio with su pi --login --shell /bin/sh --command 'pulseaudio --exit-idle-time=-1 --daemonize' and then sudo chmod a+x pulseaudio
  29. Make /etc/network/if-down.d/pulseaudio with killall pulseaudio and then sudo chmod a+x pulseaudio
  30. FIXME necessary? Add pi to pulse groups sudo usermod -aG pulse,pulse-access pi per http://askubuntu.com/questions/28176/how-do-i-run-pulseaudio-in-a-headless-server-installation

sed fanciness

March 25, 2015
bullet

Multi-line sed matching! This is easy if the pattern is only two lines. Sed line-based, so you have to issue a command to join the lines in sed's memory so you can match over the line boundaries. For example, if you want to match:

FOO BAR

First, we need to tell sed to join the lines with the N command, then we need to match on the pattern FOO\nBAR. To print the matching lines, we use the p command:

sed -n 'N; /FOO\nBAR/p' file

This works great if FOO happens to land on an even-numbered row, but fails otherwise. Why? sed only reads each line once, so it'll read line 1, then as a response to the N command read in line two............

installing gecode on ubuntu

September 7, 2012
bullet
apt-get install libgecode32 libgecode-doc libgecode-dev Alternatively, build from source, then run: ldconfig -v so that the shared libraries will be found.

standard ml on debian

September 6, 2012
bullet

There's several sml compilers in the debian repositories. My class requires polyml:

apt-get install polyml

If you want to use emacs as your editor, you may also want integrated sml support:

apt-get install sml-mode

building xfwm4-tiling on ubuntu

October 18, 2011
bullet
ubuntu, xfce, xfwm
apt-get source xfwm apt-get build-dep xfwm4 wget http://aur.archlinux.org/packages/xfwm4-tiling/xfwm4-tiling.tar.gz tar -xvvzf xfwm4-tiling.tar.gz patch -p0 < xfwm4-tiling/xfwm4-4.8.2-tiling-1.patch cd xfwm4-4.8.2/ dpkg-buildpackage -rfakeroot -uc -us -nc -b cd .. sudo dpkg -i xfwm4_4.8.2-0ubuntu1_i386.deb

dynamic dns with djbdns

June 24, 2011
bullet
djbdns, dyndns, ssh, tinydns

First set up djbdns (debian-flavored!):1

apt-get install dbndns

Create users:

adduser --no-create-home --disabled-login --shell /bin/false dnslog adduser --no-create-home --disabled-login --shell /bin/false tinydns

Make sure tinydns is linked in the right place:

cd /etc/service ; ln -sf /etc/tinydns/

Set up tinydns and check that it's running:

MYIPADDRESS=1.2.3.4 tinydns-conf tinydns dnslog /etc/tinydns/ $MYIPADDRESS svstat /etc/service/tinydns svc -d /etc/service/tinydns svc -u /etc/service/tinydns

Configure tinydns to answer nameserver requests for the subdomain:2

cd /service/tinydns/root ./add-ns dyn.servername.com $MYIPADDRESS

Tinydns will now respond to dns requests on a.ns.dyn.servername.com. Also add the subdomain as a normal host:

./add-host dyn.servername.com $MYIPADDRESS

And finally compile tinydns' data file:

make

Next, the dynamic domain needs to be registered and delegated to the correct IP address. Zone file:

dyn 86400 IN NS a.ns.dyn.servername.com. dyn 86400 IN A MYIPADDRESS a.ns.dyn 86400 IN A MYIPADDRESS

This tells clients to check with a.ns.dyn.servername.com for all requests heading to a subdomain of dyn.servername.com. I'm not sure if the dyn 86400 IN A MYIPADDRESS rule is necessary, since the same subdomain is registered with tinydns. After these changes propagate, tinydns should answer all requests to *.dyn.servername.com.

Next the computer with a dynamic IP address needs a way to update tinydns. As root, create dyn-update.sh in /etc/service/tinydns/root/:

#!/bin/bash cd /etc/service/tinydns/root IP=$1 HOST=$2 sed -i "/${HOST}/d" data echo "Setting ${HOST}.dyn.servername.com to ${IP}" ./add-alias "${HOST}.dyn.servername.com" $IP sed -i "/${HOST}/{s|86400$|300|g}" data make

The script takes two parameters: an IP address and a hostname; it removes lines matching HOST (anywhere on the line, so don't reuse-hostnames!) and sets HOST.dyn.server.name.com to the given IP.

Create a new user as whom the dynamic server can connect:

adduser dyn-update

And give them NOPASSWD sudo rights to the update script:

Cmnd_Alias DYNUPDATE=/etc/service/tinydns/root/dyn-update.sh dyn-update ALL=(ALL) NOPASSWD: DYNUPDATE

As auser On the dynamic machine, create a password-free public key, and put it on the dns host:

ssh-keygen cat .ssh/id_rsa.pub | ssh dyn-update@dyn.servername.com 'cat >> .ssh/authorized_keys'

Finally, as root on the dynamic machine, create dyn-update in /etc/network/if-up.d/:

#!/bin/bash su -c 'ssh dyn-update@dyn.servername.com '\''IP=`echo $SSH_CLIENT | sed "s| .*$||g"`; sudo /etc/service/tinydns/root/dyn-update.sh $IP mydynamichostname'\' auser

This su's to auser and then executes a command as dyn-update on dyn.servername.com. The command fetches the current IP address of the dynamic machine from SSH_CLIENT, then passes this as a parameter to dyn-update.sh along with mydynamichostname, effectively registering mydynamichostname.dyn.servername.com.

AND WE'RE DONE!

updating imdb script

November 7, 2010
bullet
imdb, mysql, python
new import script alter table movies add (imdb_title varchar(255) not null); years < 1901 don't work alter table movies drop column year; alter table movies add column (year smallint); want an index on imdb_title: drop index movie_titles on movies; create index movie_titles on movies (imdb_title, title); want genres: create table genres ( id int not null auto_increment primary key, movie_id int not null, genre varchar(12) not null ); want countries: create table countries ( id int not null auto_increment primary key, movie_id int not null, country varchar(70) not null ); much better and faster search field creation: insert into movies_search(movie_id, search_field) select m.id, concat(m.imdb_title, " ", ifnull(dirnames.names, ""), " ", ifnull(alttitles.titles, "")) as search_field from movies m left outer join alttitles on m.id = alttitles.movie_id left outer join dirnames on m.id = dirnames.movie_id;

lines only in a single file

October 25, 2010
bullet
comm

With sorted file1 and file2, shows lines in file1 which have no corresponding line in file2:

comm -2 -3 file1 file2

selective field printing with awk

October 18, 2010
bullet
awk

You have a tab-delimited file and only want certain fields.

awk 'BEGIN{OFS=FS="\t"}{print $3,$4 }' ipa_uni_test.txt

Important things:

  • FS is the "field separator", so it could be "," or similar instead of "\t"
  • OFS is the "output field separator". This replaces the ',' in the print statement.
  • $3 is a positional variable; it's the 3rd field.