This is the assumptions of John Searle's thought experiment entitled "The Chinese Box". Published in Behavioral and Brain Sciences in 1980, Searle asks "if a machine can simulate an intelligent conversation, is the machine understanding thought?" According to Searle, the machine could present the highest level of intelligence but the machine would still understand nothing. It's intelligence would strictly be bound to the program operating the machine- a set of defined parameters of responses. There would be no comprehension and thus no understanding. The machine would be nothing more than a machine.
Treading down this line of thought tends to get a bit like science fiction, but these sorts of philosophical thought experiments are becoming testable. It is obvious that technology evolves exponentially. There is now more space and speed within our machines than ever before. Artificial Intelligence is right around the corner. The important thing to remember is that this new intelligence that we as a species are creating is artificial. It will never be able to understand, comprehend, feel, or empathize. This new intelligence, regardless of its packaging, will always taste artificial.
Now certainly no one has the psychic insight to divine our future evolutionary relationships with the machines that we create, but there is high probability that our future machines will be biologically integrated within the human body; it is inevitable. Though some may fear this type of transition, it will eventually be common place. As homoseipiens, it has been our tools and technology that have spurred our evolution. Our modern situation is no different. We are changing what it means to be human with the tools and technologies that we create and develop. Only when we have fully incorporated biologic machines and the human mind will the delineation between man and machine be forever blurred beyond distinction.
