Analyzing IBM PHD Heap Dumps with Eclipse MAT   no comments

Posted at 10:49 am in Uncategorized

I struggled to get Eclipse Memory Analyzer to open the IBM phd format heap dumps, but not because it’s hard, just because it’s poorly documented. The document I was working from that best described the process was http://www.ibm.com/developerworks/java/jdk/tools/dtfj.html, which was very poor.

In brief:

  • open MAT, go to Help -> Install New Software
  • Enter http://public.dhe.ibm.com/software/websphere/runtimes/tools/dtfj/ in the “Work with” field.
  • “Add…”, call it whatever you like.
  • Diagnostic Tool Framework for Java is listed under the category “IBM Runtime Monitoring and Diagnostic Tools”.
  • Keep going, it’s fairly obvious from this point.

Further to this, if you want to analyse a large heap dump (one that causes a java heap error when you try to open it), there are instructions here. You can change the 2g to 4g or however many Gigabytes you need it to be.

Tom

Written by Tom on November 24th, 2014

Magento python additional_attributes   no comments

Posted at 3:12 pm in Uncategorized

After much hunting around I finally managed to get additional_attributes feeding
in to Magento using the suds SOAP module in python.
There are three important pieces of information I needed to get it working:

  • there needs to be a ‘single_data’ wrapper around the attribute list.
  • you need to manually use ‘associativeEntity’, suds doesn’t interpret the data structure correctly in this situation.
  • you need to create each ‘associativeEntity’ and append it to the list.

I got some of the info from this stackoverflow article which uses php.

The actual code is something like:

additional_attributes = {'single_data': []}
entity = client.factory.create('associativeEntity')
entity.key = 'attribute_name'
entity.value = 'Attribute Value'
additional_attributes['single_data'].append(entity)

There's also 'multi_data', though I haven't had need for it as yet so I don't know if or how it differs.

HTH!

Tom

Written by Tom on September 30th, 2014

Getting the Magento SOAP API working with Python   no comments

Posted at 9:02 am in Uncategorized

Getting the API working was surprisingly difficult, here are a few tips I hope will help others save a lot of wasted hours!

  • Turn off caching until you’ve got things working. It may make the site return faster, but the caching will confuse you by making the results of any changes you make appear pretty much random. Once you’ve got it working, turn it on – it really is much faster once cached. If you decide to keep the caching turned on you can delete the contents of magento/var/cache/ when you want to clear it out.
  • There are two types of user, the API needs users/roles setting up in the Users and Roles parts of System->Web Services. First set up a role with the appropriate resource access permissions, then create a user and assign the role to that user.
  • Ensure you use the correct host name in all places (apache config, magento config, browser) as using the wrong one can cause the API to fail.
  • System->Developer->Log Settings is currently a waste of time. I tried for some while to get any output at all from an error I was receiving (‘Invalid data given. Details in error message’). I have no idea where these details were, but they really weren’t in the error message. For reference, this was caused by using an incorrect Attribute Set ID – I’d stupidly assumed it would be zero, but apparently the default Attribute Set ID is 4.
  • The WSDL URI can be one of a number of possible URIs – from reading around there were something like 3 different ones in use by different people, more than one may work for you. Leaving off the first part of the URL (‘http://www.hostname.com/magento’ or similar), from what I can remember they were something like: ‘/index.php/api/v2_soap/?wsdl’ and ‘/api/v2_soap/?wsdl’. I’ve forgotten the others. You’ll need to ensure the apache rewrite module is enabled and the .htaccess file is present in the magento directory. In .htaccess you’ll need to remove the comment characters from the beginning of the RewriteBase line and the api RewriteRule line. You may also have to add ‘^index.php’ to the beginning of the api query, this will depend on your setup, I won’t explain it here as it gets complicated. Trust me, if it involves RewriteRules it’s complicated – even if it’s simple. I’m sure I’ve seen the joke, ‘Mary solved her problem by using RewriteRules, now she has 2 problems’. Don’t get me wrong, they’re invaluable, but even with logging turned on it can be hard to work out why it’s not doing what you expected it to do. Also, if you decide to debug using RewriteLog be aware that it does not work in .htaccess, it has to be in a VirtualHost or the main config somewhere.

I used the python suds module for SOAP communication. I tried a number of others but this seemed like the best at the current time (it may be just me but the actual ‘magento’ modules actually made it more difficult to work out why my script was failing by hiding more).

My basic example for logging in to the API and listing the products currently in Magento:


#!/usr/bin/python

wsdl_file = 'http://localmagento/magento/index.php/api/v2_soap/?wsdl'
magento_user = 'username' # The user you created in System->Web Services
magento_password = 'password'

import suds
from suds.client import Client

api = Client(wsdl_file)
session = api.service.login(magento_user, magento_password)
products = api.service.catalogProductList(session)
for product in products:
  print product.product_id, product.name

That covers most of what I learnt from getting it working, if I think of anything else I’ll add it here.
Feel free to ask for details if I’ve missed anything that might help. It would make a nice change from ‘I saw your site and decided to try and advertise my viagra/SEO/shoes or whatever in your comments section’.

Tom

Written by Tom on June 11th, 2014

Debian preseed installation help   no comments

Posted at 10:10 am in Uncategorized

I was caught by a couple of things trying to get initrd preseeding to work.

You should now use genisoimage instead of mkisofs.

When I copied the files from the iso I downloaded I missed the ‘.disk’ directory as it’s hidden. This caused the install to fail with “the cd-rom drive contains a cd which cannot be used for installation”.

Also when I copied the files the first time I think I managed to mount the image with incorrect switches as somehow it had ignored the rock ridge extensions and copied every file in 8.3 format, and all lower-case. When I then created the iso from that it was of course completely wrong.

Useful instructions for creating an iso with preseed file included in the iso.

Also, to make the initial menu timeout so it will automatically skip it and begin the install, add ‘timeout 50′ and ‘totaltimeout 50′ to menu.cfg.

I struggled for some time to get grub to preseed, it kept stopping at various points (and of course it’s the last thing so every change I made required testing by a complete reinstall, making for a very slow test cycle). In the end I found that it’s necessary to add:

d-i grub-installer/only_debian boolean true
d-i grub-installer/with_other_os boolean true
d-i grub-pc/install_devices multiselect /dev/vda

to the preseed.cfg file, ensuring it installs grub without asking and specifying the device to install grub onto as /dev/vda (the virtual disk device name using virt-install in libvirt and kvm).

Tom

Written by Tom on May 29th, 2013

Installing KVM virtualisation under Debian Wheezy   no comments

Posted at 12:21 pm in Uncategorized

Following the instructions I found on Howtoforge here is my experience and any additional information others might find useful. You know, the little gotchas that didn’t come up for people that already knew what they were doing when they wrote the install guide.

Having found my cpu does indeed support kvm the “modprobe kvm_intel” failed due to “operation not supported”. A quick google led me to an additional command to run to find out why it’s not supported:

dmesg | grep kvm

Which told me:

disabled in bios

More googling later told me how to enable it in HP bios (confusingly – to me at least – it’s under the Security menu). Apparently some bios’ don’t have any option to enable it, in which case you’re stuck unless you’re willing to flash a new bios that does support it.

During the:

virsh -c qemu:///system list

step there was an error with being unable to connect to the gnome-keyring daemon:

WARNING: gnome-keyring:: couldn't connect to: /tmp/keyring-S31tpg/pkcs11: No such file or directory

This was fixed using comment 18 in a thread on redhat bugzilla. I’d switched to XFCE and XFCE was not in the list. After adding it to the list and logging out and back in again the virsh command worked just fine.

As an aside – I was amused if not actually helped by, “If it displays an error instead, then something went wrong.”

My networking knowledge is less than perfect, so I can now tell you that the change of eth0 from:

auto eth0
iface eth0 inet static

to:

auto eth0
iface eth0 inet manual

is entirely deliberate and necessary.

Basically what you want to end up with is br0 configured almost identically to how eth0 was configured only with the additional “bridge_*” options. In other words if eth0 was using dhcp before, br0 should now use dhcp. If it’s configured on a static IP, br0 should be configured on that IP instead.

The next step I had a problem with was using virt-install to install a debian wheezy iso. My partitioning schema provided only a couple of Gb to the partition with /var/lib/libvirt/images on. To work around this I moved the images directory to a larger partition and symlinked /var/lib/libvirt/images to there.

Following that I was ready to use virt-manager. Confusingly I’d managed to get to this step without installing virt-manager and don’t remember a step saying to. I probably missed it somehow, but regardless, at this point I installed it.

Once in virt-manager I didn’t have to do any of the steps to make vm10 visible, it was already there. Presumably the software has improved since the instructions were written. vm10 was paused, opening it revealed a very colourful and crashed screen. I don’t know how it got to that state but I was required to force power-off to vm10, after which it attempted to boot from cd which it couldn’t, and then pxe boot which it also couldn’t, after which it decided it had no more bootable devices, which seems fair if it hasn’t had a chance to install yet.

Somehow I’d downloaded the debian iso for the wrong architecture – ia64 instead of amd64. Downloading the correct iso helped enormously, as you might expect.

The rest of the tutorial was unnecessary for me as I only wanted to get to the installed point, and then only via command-line not GUI, so I have not covered these.

Tom

Written by Tom on May 28th, 2013

Booting from a USB key that won’t boot   no comments

Posted at 8:36 am in Uncategorized

I recently had a USB thumb drive that refused to boot so that I could install a new OS from it. After most of a day of trawling the internet looking for answers I finally managed to get it working.

The drive in question was a Sandisk Cruzer (16Gb) though as far as I can tell the final solution may work for other non-booting USB keys.

Firstly, I removed the U3 partition (I’m not sure this was necessary in the end, but you may find it helps). You may not have a U3 partition, in which case you can skip removing it. Instructions for removal are on the sandisk site. TIP: This will not run under later versions of windows, failing silently – the trick is to modify it to run in Windows XP SP2 compatibility mode.

After much fiddling I finally found dummy910 on the ubuntu forums saying to format the drive as fat16 rather than fat32. How you go about formatting it as fat16 depends on what OS you have. I have access to Windows 7 and OSX, finding it easier to format it under OSX. If you want to do it under another OS I’ll leave it to you to do the googling.

Due to issues with FAT16 and sizing the actual command I used was:

diskutil partitionDisk /dev/disk1 2 MBRFormat "MS-DOS FAT16" "DATA" 512M "MS-DOS FAT32" "DATA2" 15000M

 

This makes the first 512Mb of the drive FAT16 and the next 15Gb FAT32. For reasons I’ve not looked into if I attempted to format just the FAT16 partition alone it would complain about the size – also formatting the FAT32 partition allowed it to work.

After formatting as fat16 I applied the boot images to the USB key (a googling I’ll also leave to you). Unless you’re trying to install a linux distro from a USB key, in which case you can use UNetbootin as I did.

Tom

Addendum: Having done all of the above I reformatted the whole thing as one FAT32 partition (through the Erase option on Disk Utility on OSX) and tried again, and this time it actually worked. Mystifying, hand-wavy and possibly closer to voodoo than technical. I can only assume one of the above steps changed something on the USB drive. YMMV.

Written by Tom on May 24th, 2013

Creating a Nautilus shell in Blender   no comments

Posted at 9:12 pm in Uncategorized

After seeing pictures of the Nautilus house near Mexico City I was inspired to attempt to replicate something like it in Blender as a learning exercise. After trying the Screw Modifier and Screw Tools I couldn’t make them do quite what I wanted so I came up with another way.

Starting with a blank scene, create a curve (Add->Curve->Bezier). With the curve selected, change the Transform->Rotation->Y to 90 (I’m working with this because that constructs the shell at the same angle as the Nautilus house).

You’ll find the next bit easier in Orthographic view looking along X-axis (3 on the numberpad).

01_Curve_Edit

Tab into Edit mode and select both end points of the curve, go to Curve->Segments->Subdivide to add an additional point in the middle.

Select the middle point only, then translate it out to where you want the widest point of the shell to be, followed by adjusting the handles on the points to achieve the curve you want for the shell. Keep the two end points on the origin (x=0,y=0) as the rotation to create the shell occurs around that.

02_Curve_Edit2

Tab back to Object mode; with the curve selected press Alt+C for the ‘Convert To’ menu, select ‘Mesh from Curve’. Now Tab to Edit mode and you’ll see that the curve is now a Mesh. Select all of the points on the curve.

03_Curve_Mesh

In the Screen dropdown (at the top, currently set to Default) select Scripting. Change the view to directly above looking down (7 on the numberpad).

In the grey box on the left, click ‘+ New’, then copy and paste the below script into the box:

import bpy
bpy.ops.object.editmode_toggle()
for i in range(32):
  bpy.ops.mesh.extrude_region_move()
  bpy.ops.transform.rotate(value=0.145907, axis=(0, 10, 1), constraint_axis=(False, False, True), release_confirm=False)
  bpy.ops.transform.resize(value=(0.976289, 0.976289, 0.976289), constraint_axis=(True, True, False), constraint_orientation='GLOBAL', mirror=False, proportional='DISABLED', proportional_edit_falloff='SMOOTH', proportional_size=1, snap=False, snap_target='CLOSEST', snap_point=(0, 0, 0), snap_align=False, snap_normal=(0, 0, 0), texture_space=False, release_confirm=False)

04_Scripting

At this point I discovered that if you run the script as is using the ‘Run Script’ button blender will rotate around the median of the object. See http://projects.blender.org/tracker/?func=detail&atid=498&aid=31640&group_id=9 for a discussion about why. All I can say is it would be nice to be able to rotate about a specified pivot point given in the script or refer to one in the view, it seems a bit arbitrary as it is. I can’t see a way to make it work without serious kludging.

In order to move the median point to the origin you have to create more points on the other side of the origin to balance it out.

05_Kludging_The_Median

What I’ve done above is apply a mirror modifier to exactly balance the points, then because we don’t need to care about the vertical rotation grabbed the bottom half of the new points and moved them down and the upper half and moved them up (It allows us to more easily separate them from the rest of the model later).

06_Silo_Thing

Deselect all of the points, then select the extra points as above using the box tool. Now delete them.

You should be left with the shell. It’s not perfect by any means but it’s a start. I’m currently thinking of alternative ways to do the same thing only better, with more control over the final form, if I can work around the rotation about the median issue.

07_Shell

 

And here’s one with coloured windows added:

nautilus-house

The wall and windows were fairly complicated for me and were done by using the boolean operator to make the holes for the windows.

Written by Tom on February 5th, 2013

Tagged with , , , , ,