Widely available digital ophthalmology data can be used to implement accurate Computer-Aided Diagnosis Systems. In this article we describe an automatic system which combines text clinical annotations, demographical information, as well as different types of ophthalmology image data to issue a recommendation for cataract surgery. Textual annotations are encoded using a standardized medical ontology nomenclature to enable higher level modeling. Image data is processed by convolutional neural networks to extract compact features. These two types of data together with demographical information are then inputted into a random forest classifier which then decides if surgery is recommended. The method proposed is evaluated on a real-life dataset, achieving accuracies and precisions around 90%. Several conclusions are drawn concerning the usefulness of the different input data types, used independently or combined.