The number of smart homes is increasingly expanding, with even more connected devices and available control options. Mobile devices have unfortunately been up to now generally regarded as mere remote controls in these environments. This paper addresses this shortcoming, by presenting a novel integration architecture and prototype where the potential of mobile devices sensors can be better explored in home automation platforms, in particular by detecting objects in the information collected by their cameras that subsequently allow for users to interact with them in an intuitive way. The detection is performed at the mobile side, using a lightweight machine learning solution. The obtained accuracy and processing time are comparable to that obtained at server side. But the advantage here is that the interactive experience of users can be dramatically improved, with the absence of round-trip time required if server processing would be used.